* [PATCH v1 00/12] f2fs: introduce metadata cache
@ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel
0 siblings, 0 replies; 34+ messages in thread
From: Chao Yu @ 2026-08-20 3:17 UTC (permalink / raw)
To: jaegeuk; +Cc: linux-f2fs-devel, linux-kernel, Chao Yu
This patchset introduces a self-managed metadata block cache in f2fs,
decoupling meta blocks, node blocks, and compressed data blocks from
the Linux VFS page cache and fake internal inodes.
=== 1. Background & Motivation ===
Currently, F2FS uses fake VFS inodes (meta_inode, node_inode, and
compress_inode) to manage internal block caching through the VFS page
cache. Because of this implementation, the f2fs block size was
historically coupled to the kernel page size.
We now want to unbind block size from page size to support configurations
where block size <= PAGE_SIZE (e.g., mounting a 4KB-block F2FS image on a
16KB or 64KB page system).
One possible approach is to continue using the VFS page cache to store
metadata blocks. However, doing so introduces three major architectural
issues (illustrated by a 4KB block on a 16KB page system):
1. Memory Overhead:
Metadata access patterns are typically random and sparse. Caching a
single 4KB metadata block inside a page cache folio forces the
allocation of an entire 16KB folio, resulting in 4x memory waste.
2. Folio and Sub-block Conversion Complexity:
Using larger folios requires tracking individual sub-block dirty/uptodate
states within each folio and performing index-to-offset conversions across
function boundaries. Because core metadata structures (e.g., f2fs_checkpoint,
f2fs_sit_block, f2fs_nat_block, f2fs_summary_block, f2fs_node) are accessed
extensively throughout the filesystem, this sub-block management and offset
calculation complexity would spread across the entire F2FS codebase.
3. Lock Contention:
Multiple independent node blocks (e.g., dnode blocks belonging to
different files) can reside within the same folio. Concurrent fsync()
calls on unrelated files would contend on the same folio_lock(),
serializing metadata updates and degrading multi-threaded performance.
Decoupling metadata caching from PAGE_SIZE by allocating exact
block-sized cache entries is the critical first step toward supporting
4KB-block F2FS images on 16KB/64KB page systems.
=== 2. Metadata Cache Architecture & Design ===
This patchset introduces a dedicated, block-size-aligned caching
3. Lock Contention:
Multiple independent node blocks (e.g., dnode blocks belonging to
different files) can reside within the same folio. Concurrent fsync()
calls on unrelated files would contend on the same folio_lock(),
serializing metadata updates and degrading multi-threaded performance.
3. Lock Contention:
Multiple independent node blocks (e.g., dnode blocks belonging to
different files) can reside within the same folio. Concurrent fsync()
calls on unrelated files would contend on the same folio_lock(),
serializing metadata updates and degrading multi-threaded performance.
Decoupling metadata caching from PAGE_SIZE by allocating exact
block-sized cache entries is the critical first step toward supporting
4KB-block F2FS images on 16KB/64KB page systems.
=== 2. Metadata Cache Architecture & Design ===
This patchset introduces a dedicated, block-size-aligned caching
infrastructure with the following key components:
- Block-Size Aligned Allocation:
Allocates memory buffers matching exactly the filesystem block size
(4KB or 16KB) via kzalloc(), fully independent of the host
architecture's PAGE_SIZE.
- Radix Tree Indexing with Fast Tag Scanning:
Each cache instance (META_CACHE, NODE_CACHE, COMPRESS_CACHE) indexes
cached blocks via a radix tree (keyed by Physical Block Address for meta/
compress cache, and Node ID for node cache). Radix tree tags
(F2FS_CACHE_TAG_DIRTY, F2FS_CACHE_TAG_WRITEBACK) provide O(1) batch gang
lookups for flushing and writeback without dual-list shuffling.
- Lightweight Bit-Locking:
Individual entries use atomic bit locks (F2FS_BLOCK_LOCKED via
wait_on_bit_lock() / clear_and_wake_up_bit()) rather than heavyweight
embedded mutexes/semaphores, minimizing memory footprint per entry.
- Direct BIO Read/Write & BIO Merging:
Decouples metadata/node I/O from VFS address spaces by submitting direct
BIOs (f2fs_submit_cache_read / f2fs_submit_cache_write) with chained
adjacent vector merging (entry->next_entry) and dedicated completion
handlers.
- Memory Reclamation Shrinker:
Integrates with the kernel shrinker subsystem via a 3-phase isolation
algorithm (isolate unreferenced clean entries -> truncate from radix tree
under lock -> splice un-reclaimed entries back to LRU) to safely reclaim
clean cached blocks under system memory pressure.
- Background Writeback Kthread & Checkpoint Integration:
Provides a dedicated background kthread (f2fs_writeback-X:Y) for periodic
dirty cache flushing, combined with synchronous flushing during checkpoint
commit.
- Fault Injection, Tracepoints & Debugfs Observability:
Integrates FAULT_KALLOC fault injection, tracepoints for cache state
transitions and batch writeback, and per-cache memory breakdowns in debugfs.
=== 3. Patchset Organization ===
- Patch 01: Implement the core metadata cache infrastructure & direct BIO I/O.
- Patch 02: Initialize and teardown META_CACHE in sb_info.
- Patch 03: Integrate metadata cache into the memory shrinker subsystem.
- Patch 04: Introduce the background writeback kernel thread.
- Patch 05: Migrate metadata block caching (SIT, NAT, SSA, CP, recovery, GC)
from meta_inode to META_CACHE.
- Patch 06: Initialize and teardown NODE_CACHE in sb_info.
- Patch 07: Migrate node and inode block caching from node_inode to NODE_CACHE.
- Patch 08: Initialize and teardown COMPRESS_CACHE in sb_info.
- Patch 09: Migrate compressed cluster caching from compress_inode to COMPRESS_CACHE.
- Patch 10: Add fault injection support for cache allocation paths.
- Patch 11: Introduce ftrace tracepoints for cache dirty and writeback events.
- Patch 12: Expose per-cache memory usage in debugfs.
Chao Yu (12):
f2fs: cache: implement metadata cache
f2fs: cache: initialize meta cache
f2fs: cache: introduce shrinker
f2fs: cache: introduce writeback thread
f2fs: cache: use meta cache
f2fs: cache: initialize node cache
f2fs: cache: use node cache
f2fs: cache: initialize compress cache
f2fs: cache: use compress cache
f2fs: cache: support fault injection
f2fs: cache: introduce tracepoints
f2fs: cache: show per-cache usage in debugfs
fs/f2fs/Makefile | 2 +-
fs/f2fs/acl.c | 26 +-
fs/f2fs/acl.h | 8 +-
fs/f2fs/cache.c | 690 +++++++++++++++++++++++++
fs/f2fs/cache.h | 224 ++++++++
fs/f2fs/checkpoint.c | 404 +++++++--------
fs/f2fs/compress.c | 171 +++----
fs/f2fs/data.c | 527 +++++++++++++------
fs/f2fs/debug.c | 70 ++-
fs/f2fs/dir.c | 168 +++---
fs/f2fs/extent_cache.c | 14 +-
fs/f2fs/f2fs.h | 333 ++++++------
fs/f2fs/file.c | 78 ++-
fs/f2fs/gc.c | 178 ++++---
fs/f2fs/inline.c | 284 ++++++-----
fs/f2fs/inode.c | 205 +++-----
fs/f2fs/iostat.h | 11 +
fs/f2fs/namei.c | 118 ++---
fs/f2fs/node.c | 994 +++++++++++++++++-------------------
fs/f2fs/node.h | 109 ++--
fs/f2fs/recovery.c | 253 ++++-----
fs/f2fs/segment.c | 263 +++++-----
fs/f2fs/segment.h | 39 +-
fs/f2fs/shrinker.c | 13 +
fs/f2fs/super.c | 143 +++---
fs/f2fs/xattr.c | 123 +++--
fs/f2fs/xattr.h | 12 +-
include/linux/f2fs_fs.h | 3 -
include/trace/events/f2fs.h | 71 +++
29 files changed, 3331 insertions(+), 2203 deletions(-)
create mode 100644 fs/f2fs/cache.c
create mode 100644 fs/f2fs/cache.h
--
2.49.0
^ permalink raw reply [flat|nested] 34+ messages in thread* [f2fs-dev] [PATCH v1 00/12] f2fs: introduce metadata cache @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-kernel, linux-f2fs-devel This patchset introduces a self-managed metadata block cache in f2fs, decoupling meta blocks, node blocks, and compressed data blocks from the Linux VFS page cache and fake internal inodes. === 1. Background & Motivation === Currently, F2FS uses fake VFS inodes (meta_inode, node_inode, and compress_inode) to manage internal block caching through the VFS page cache. Because of this implementation, the f2fs block size was historically coupled to the kernel page size. We now want to unbind block size from page size to support configurations where block size <= PAGE_SIZE (e.g., mounting a 4KB-block F2FS image on a 16KB or 64KB page system). One possible approach is to continue using the VFS page cache to store metadata blocks. However, doing so introduces three major architectural issues (illustrated by a 4KB block on a 16KB page system): 1. Memory Overhead: Metadata access patterns are typically random and sparse. Caching a single 4KB metadata block inside a page cache folio forces the allocation of an entire 16KB folio, resulting in 4x memory waste. 2. Folio and Sub-block Conversion Complexity: Using larger folios requires tracking individual sub-block dirty/uptodate states within each folio and performing index-to-offset conversions across function boundaries. Because core metadata structures (e.g., f2fs_checkpoint, f2fs_sit_block, f2fs_nat_block, f2fs_summary_block, f2fs_node) are accessed extensively throughout the filesystem, this sub-block management and offset calculation complexity would spread across the entire F2FS codebase. 3. Lock Contention: Multiple independent node blocks (e.g., dnode blocks belonging to different files) can reside within the same folio. Concurrent fsync() calls on unrelated files would contend on the same folio_lock(), serializing metadata updates and degrading multi-threaded performance. Decoupling metadata caching from PAGE_SIZE by allocating exact block-sized cache entries is the critical first step toward supporting 4KB-block F2FS images on 16KB/64KB page systems. === 2. Metadata Cache Architecture & Design === This patchset introduces a dedicated, block-size-aligned caching 3. Lock Contention: Multiple independent node blocks (e.g., dnode blocks belonging to different files) can reside within the same folio. Concurrent fsync() calls on unrelated files would contend on the same folio_lock(), serializing metadata updates and degrading multi-threaded performance. 3. Lock Contention: Multiple independent node blocks (e.g., dnode blocks belonging to different files) can reside within the same folio. Concurrent fsync() calls on unrelated files would contend on the same folio_lock(), serializing metadata updates and degrading multi-threaded performance. Decoupling metadata caching from PAGE_SIZE by allocating exact block-sized cache entries is the critical first step toward supporting 4KB-block F2FS images on 16KB/64KB page systems. === 2. Metadata Cache Architecture & Design === This patchset introduces a dedicated, block-size-aligned caching infrastructure with the following key components: - Block-Size Aligned Allocation: Allocates memory buffers matching exactly the filesystem block size (4KB or 16KB) via kzalloc(), fully independent of the host architecture's PAGE_SIZE. - Radix Tree Indexing with Fast Tag Scanning: Each cache instance (META_CACHE, NODE_CACHE, COMPRESS_CACHE) indexes cached blocks via a radix tree (keyed by Physical Block Address for meta/ compress cache, and Node ID for node cache). Radix tree tags (F2FS_CACHE_TAG_DIRTY, F2FS_CACHE_TAG_WRITEBACK) provide O(1) batch gang lookups for flushing and writeback without dual-list shuffling. - Lightweight Bit-Locking: Individual entries use atomic bit locks (F2FS_BLOCK_LOCKED via wait_on_bit_lock() / clear_and_wake_up_bit()) rather than heavyweight embedded mutexes/semaphores, minimizing memory footprint per entry. - Direct BIO Read/Write & BIO Merging: Decouples metadata/node I/O from VFS address spaces by submitting direct BIOs (f2fs_submit_cache_read / f2fs_submit_cache_write) with chained adjacent vector merging (entry->next_entry) and dedicated completion handlers. - Memory Reclamation Shrinker: Integrates with the kernel shrinker subsystem via a 3-phase isolation algorithm (isolate unreferenced clean entries -> truncate from radix tree under lock -> splice un-reclaimed entries back to LRU) to safely reclaim clean cached blocks under system memory pressure. - Background Writeback Kthread & Checkpoint Integration: Provides a dedicated background kthread (f2fs_writeback-X:Y) for periodic dirty cache flushing, combined with synchronous flushing during checkpoint commit. - Fault Injection, Tracepoints & Debugfs Observability: Integrates FAULT_KALLOC fault injection, tracepoints for cache state transitions and batch writeback, and per-cache memory breakdowns in debugfs. === 3. Patchset Organization === - Patch 01: Implement the core metadata cache infrastructure & direct BIO I/O. - Patch 02: Initialize and teardown META_CACHE in sb_info. - Patch 03: Integrate metadata cache into the memory shrinker subsystem. - Patch 04: Introduce the background writeback kernel thread. - Patch 05: Migrate metadata block caching (SIT, NAT, SSA, CP, recovery, GC) from meta_inode to META_CACHE. - Patch 06: Initialize and teardown NODE_CACHE in sb_info. - Patch 07: Migrate node and inode block caching from node_inode to NODE_CACHE. - Patch 08: Initialize and teardown COMPRESS_CACHE in sb_info. - Patch 09: Migrate compressed cluster caching from compress_inode to COMPRESS_CACHE. - Patch 10: Add fault injection support for cache allocation paths. - Patch 11: Introduce ftrace tracepoints for cache dirty and writeback events. - Patch 12: Expose per-cache memory usage in debugfs. Chao Yu (12): f2fs: cache: implement metadata cache f2fs: cache: initialize meta cache f2fs: cache: introduce shrinker f2fs: cache: introduce writeback thread f2fs: cache: use meta cache f2fs: cache: initialize node cache f2fs: cache: use node cache f2fs: cache: initialize compress cache f2fs: cache: use compress cache f2fs: cache: support fault injection f2fs: cache: introduce tracepoints f2fs: cache: show per-cache usage in debugfs fs/f2fs/Makefile | 2 +- fs/f2fs/acl.c | 26 +- fs/f2fs/acl.h | 8 +- fs/f2fs/cache.c | 690 +++++++++++++++++++++++++ fs/f2fs/cache.h | 224 ++++++++ fs/f2fs/checkpoint.c | 404 +++++++-------- fs/f2fs/compress.c | 171 +++---- fs/f2fs/data.c | 527 +++++++++++++------ fs/f2fs/debug.c | 70 ++- fs/f2fs/dir.c | 168 +++--- fs/f2fs/extent_cache.c | 14 +- fs/f2fs/f2fs.h | 333 ++++++------ fs/f2fs/file.c | 78 ++- fs/f2fs/gc.c | 178 ++++--- fs/f2fs/inline.c | 284 ++++++----- fs/f2fs/inode.c | 205 +++----- fs/f2fs/iostat.h | 11 + fs/f2fs/namei.c | 118 ++--- fs/f2fs/node.c | 994 +++++++++++++++++------------------- fs/f2fs/node.h | 109 ++-- fs/f2fs/recovery.c | 253 ++++----- fs/f2fs/segment.c | 263 +++++----- fs/f2fs/segment.h | 39 +- fs/f2fs/shrinker.c | 13 + fs/f2fs/super.c | 143 +++--- fs/f2fs/xattr.c | 123 +++-- fs/f2fs/xattr.h | 12 +- include/linux/f2fs_fs.h | 3 - include/trace/events/f2fs.h | 71 +++ 29 files changed, 3331 insertions(+), 2203 deletions(-) create mode 100644 fs/f2fs/cache.c create mode 100644 fs/f2fs/cache.h -- 2.49.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH v1 01/12] f2fs: cache: implement metadata cache 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel -1 siblings, 0 replies; 34+ messages in thread From: Chao Yu @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-f2fs-devel, linux-kernel, Chao Yu This patch introduces the core metadata block caching infrastructure to manage f2fs metadata independently of the page cache. It implements: - core cache APIs: get, create, put, drop, backed by a radix tree and a single global LRU list. - support multiple status of cached block: LOCKED, UPTODATE, DIRTY, WRITEBACK, INLINE. - internal bio based read/write helpers with adjacent block vector merging. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/Makefile | 2 +- fs/f2fs/cache.c | 531 +++++++++++++++++++++++++++++++++++++++++++++++ fs/f2fs/cache.h | 179 ++++++++++++++++ fs/f2fs/data.c | 283 ++++++++++++++++++++++--- fs/f2fs/f2fs.h | 24 +++ fs/f2fs/iostat.h | 11 + 6 files changed, 1002 insertions(+), 28 deletions(-) create mode 100644 fs/f2fs/cache.c create mode 100644 fs/f2fs/cache.h diff --git a/fs/f2fs/Makefile b/fs/f2fs/Makefile index 8a7322d229e4..fbf49c30b066 100644 --- a/fs/f2fs/Makefile +++ b/fs/f2fs/Makefile @@ -3,7 +3,7 @@ obj-$(CONFIG_F2FS_FS) += f2fs.o f2fs-y := dir.o file.o inode.o namei.o hash.o super.o inline.o f2fs-y += checkpoint.o gc.o data.o node.o segment.o recovery.o -f2fs-y += shrinker.o extent_cache.o sysfs.o +f2fs-y += shrinker.o extent_cache.o sysfs.o cache.o f2fs-$(CONFIG_F2FS_STAT_FS) += debug.o f2fs-$(CONFIG_F2FS_FS_XATTR) += xattr.o f2fs-$(CONFIG_F2FS_FS_POSIX_ACL) += acl.o diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c new file mode 100644 index 000000000000..08bc658166f7 --- /dev/null +++ b/fs/f2fs/cache.c @@ -0,0 +1,531 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 Google LLC + * Author: Chao Yu <chaseyu@google.com> + */ +#include "linux/spinlock.h" +#include <linux/fs.h> +#include <linux/f2fs_fs.h> +#include <linux/radix-tree.h> +#include <linux/slab.h> +#include <linux/list.h> +#include <linux/pagemap.h> +#include <linux/kthread.h> +#include <linux/freezer.h> +#include <linux/delay.h> +#include "f2fs.h" +#include "cache.h" +#include "node.h" +#include "segment.h" + +void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, + enum page_type type) +{ + /* in case the entry was truncated or on-going shrink */ + if (!entry->cache) + return; + + if (!f2fs_cache_test_writeback(entry)) + return; + + /* submit cached bio */ + f2fs_submit_merged_write_cache(entry, type); + + wait_on_bit_io(&entry->state, F2FS_BLOCK_WRITEBACK, + TASK_UNINTERRUPTIBLE); +} + +void f2fs_cache_wait_writeback(struct f2fs_cached_block *entry) +{ + /* in case the entry was truncated or on-going shrink */ + if (!entry->cache) + return; + + f2fs_cache_wait_writeback_cond(entry, + IS_META_CACHE(entry->cache) ? META : NODE); +} + +static void f2fs_cache_update_tag(struct f2fs_cached_block *entry, unsigned int src, + unsigned int dst) +{ + struct f2fs_cached_block_list *cache = entry->cache; + unsigned long flags; + + spin_lock_irqsave(&cache->tree_lock, flags); + if (src) + radix_tree_tag_clear(&cache->root, entry->index, src); + if (dst) + radix_tree_tag_set(&cache->root, entry->index, dst); + spin_unlock_irqrestore(&cache->tree_lock, flags); +} + +bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry) +{ + struct f2fs_cached_block_list *cache = entry->cache; + + if (!f2fs_cache_test_uptodate(entry)) + f2fs_cache_set_uptodate(entry); + +#ifdef CONFIG_F2FS_CHECK_FS + if (cache->type == F2FS_NODE_CACHE && IS_INODE(cache_folio(entry))) + f2fs_inode_chksum_set(cache->sbi, cache_folio(entry)); +#endif + + if (f2fs_cache_test_dirty(entry)) + return false; + + if (!f2fs_cache_test_and_set_dirty(entry)) { + enum count_type type = IS_META_CACHE(cache) ? + F2FS_DIRTY_META : F2FS_DIRTY_NODES; + + f2fs_cache_update_tag(entry, 0, F2FS_CACHE_TAG_DIRTY); + inc_page_count(cache->sbi, type); + return true; + } + + return false; +} + +bool f2fs_clear_cache_dirty(struct f2fs_cached_block *entry) +{ + if (!f2fs_cache_test_dirty(entry)) + return false; + + f2fs_cache_clear_dirty(entry); + return true; +} + +static void __f2fs_drop_cache_dirty(struct f2fs_cached_block *entry, bool force) +{ + + struct f2fs_cached_block_list *cache = entry->cache; + enum count_type type = IS_META_CACHE(cache) ? + F2FS_DIRTY_META : F2FS_DIRTY_NODES; + + f2fs_cache_clear_uptodate(entry); + + if (!force && !f2fs_clear_cache_dirty(entry)) + return; + + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_DIRTY, 0); + dec_page_count(cache->sbi, type); +} + +void f2fs_drop_cache_dirty(struct f2fs_cached_block *entry) +{ + __f2fs_drop_cache_dirty(entry, false); +} + +void f2fs_force_clear_cache_dirty(struct f2fs_cached_block *entry) +{ + __f2fs_drop_cache_dirty(entry, true); +} + +void f2fs_start_cache_writeback(struct f2fs_cached_block *entry) +{ + f2fs_cache_set_writeback(entry); + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_DIRTY, + F2FS_CACHE_TAG_WRITEBACK); +} + +void f2fs_end_cache_writeback(struct f2fs_cached_block *entry) +{ + /* + * should call f2fs_cache_update_tag() before clearing writeback bit, + * in case f2fs_truncate_cache() set entry->cache to NULL. + */ + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_WRITEBACK, 0); + clear_and_wake_up_bit(F2FS_BLOCK_WRITEBACK, &entry->state); +} + +static int f2fs_cache_refcount(struct f2fs_cached_block *entry) +{ + return atomic_read(&entry->refcount); +} + +static void __f2fs_free_cache(struct f2fs_cached_block *entry) +{ + kfree(entry->data); + kfree(entry); +} + +static void f2fs_free_cache(struct f2fs_cached_block *entry) +{ + WARN_ON_ONCE(!list_empty(&entry->list)); + WARN_ON_ONCE(f2fs_cache_refcount(entry)); + __f2fs_free_cache(entry); +} + +void f2fs_cache_get(struct f2fs_cached_block *entry) +{ + atomic_inc(&entry->refcount); +} + +static bool f2fs_cache_put(struct f2fs_cached_block *entry) +{ + WARN_ON_ONCE(!f2fs_cache_refcount(entry)); + if (atomic_dec_and_test(&entry->refcount)) { + f2fs_free_cache(entry); + return true; + } + return false; +} + +static struct f2fs_cached_block *f2fs_create_cache( + struct f2fs_cached_block_list *cache, + unsigned long index, bool nofail) +{ + struct f2fs_cached_block *entry; + unsigned int flags = GFP_NOFS; + + if (nofail) + flags |= __GFP_NOFAIL; + + entry = kzalloc_obj(*entry, flags); + if (!entry) + return ERR_PTR(-ENOMEM); + + entry->data = kzalloc(cache->sbi->blocksize, flags); + if (!entry->data) { + kfree(entry); + return ERR_PTR(-ENOMEM); + } + + entry->index = index; + + atomic_set(&entry->refcount, 0); + entry->next_entry = NULL; + INIT_LIST_HEAD(&entry->list); + + entry->cache = cache; + + return entry; +} + +static struct f2fs_cached_block *f2fs_insert_cache( + struct f2fs_cached_block_list *cache, + unsigned long index, + struct f2fs_cached_block *new) +{ + struct f2fs_cached_block *e; + int ret; + unsigned long flags; + + ret = radix_tree_preload(GFP_NOFS | __GFP_NOFAIL); + f2fs_bug_on(cache->sbi, ret); + + spin_lock(&cache->list_lock); + spin_lock_irqsave(&cache->tree_lock, flags); + e = radix_tree_lookup(&cache->root, index); + if (!e) { + e = new; + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(e)); + + ret = radix_tree_insert(&cache->root, index, e); + f2fs_bug_on(cache->sbi, ret); + + /* radix tree referenced cache entry */ + f2fs_cache_get(e); + f2fs_bug_on(cache->sbi, !list_empty(&e->list)); + list_add_tail(&e->list, &cache->lru_list); + cache->num_entries++; + } + f2fs_cache_get(e); + spin_unlock_irqrestore(&cache->tree_lock, flags); + spin_unlock(&cache->list_lock); + radix_tree_preload_end(); + + if (new != e) { + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(new)); + __f2fs_free_cache(new); + } + + return e; +} + +struct f2fs_cached_block *f2fs_find_cache( + struct f2fs_cached_block_list *cache, + unsigned long index) +{ + struct f2fs_cached_block *entry; + unsigned long flags; + + spin_lock(&cache->list_lock); + spin_lock_irqsave(&cache->tree_lock, flags); + entry = radix_tree_lookup(&cache->root, index); + if (entry) { + f2fs_bug_on(cache->sbi, !f2fs_cache_refcount(entry)); + f2fs_cache_get(entry); + if (!list_empty(&entry->list)) + list_move_tail(&entry->list, &cache->lru_list); + + } else { + entry = ERR_PTR(-ENOENT); + } + spin_unlock_irqrestore(&cache->tree_lock, flags); + spin_unlock(&cache->list_lock); + + return entry; +} + +struct f2fs_cached_block *f2fs_grab_cache( + struct f2fs_cached_block_list *cache, + unsigned long index, int flags) + +{ + struct f2fs_cached_block *entry, *new; + bool create = flags & F2FS_CACHE_CREATE; + bool nofail = flags & F2FS_CACHE_NOFAIL; + bool lock = flags & F2FS_CACHE_LOCK; + +repeat: + entry = f2fs_find_cache(cache, index); + if (!IS_ERR(entry)) + goto found; + + if (!create) + return ERR_PTR(-ENOENT); + + new = f2fs_create_cache(cache, index, nofail); + if (IS_ERR(new)) + return new; + + entry = f2fs_insert_cache(cache, index, new); +found: + if (lock) { + f2fs_lock_cache(entry); + /* has been truncated */ + if (entry->cache != cache) { + f2fs_put_cache(entry, true); + goto repeat; + } + } + return entry; +} + +bool f2fs_trylock_cache(struct f2fs_cached_block *entry) +{ + return !test_and_set_bit(F2FS_BLOCK_LOCKED, &entry->state); +} + +void f2fs_lock_cache(struct f2fs_cached_block *entry) +{ + wait_on_bit_lock(&entry->state, F2FS_BLOCK_LOCKED, + TASK_UNINTERRUPTIBLE); +} + +void f2fs_unlock_cache(struct f2fs_cached_block *entry) +{ + clear_and_wake_up_bit(F2FS_BLOCK_LOCKED, &entry->state); +} + +bool f2fs_put_cache(struct f2fs_cached_block *entry, bool unlock) +{ + if (IS_ERR_OR_NULL(entry)) + return false; + if (unlock) + f2fs_unlock_cache(entry); + return f2fs_cache_put(entry); +} + +unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache, + struct f2fs_cached_block **entries, + pgoff_t *first_index, unsigned int max_nr, + int tag) +{ + unsigned long flags; + int nr, i; + + spin_lock_irqsave(&cache->tree_lock, flags); + nr = radix_tree_gang_lookup_tag(&cache->root, (void **)entries, + *first_index, max_nr, tag); + if (!nr) + goto out; + + for (i = 0; i < nr; i++) + f2fs_cache_get(entries[i]); + *first_index = entries[nr - 1]->index + 1; +out: + spin_unlock_irqrestore(&cache->tree_lock, flags); + return nr; +} + +void f2fs_cache_gang_release(struct f2fs_cached_block **entries, + unsigned int nr_entries) +{ + int i; + + for (i = 0; i < nr_entries; i++) + f2fs_put_cache(entries[i], false); +} + +void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache) +{ + unsigned long index = 0; + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; + int nr, i; + +next: + nr = f2fs_cache_gang_lookup_tag(cache, entries, &index, + F2FS_ONSTACK_CACHES, F2FS_CACHE_TAG_WRITEBACK); + if (!nr) + return; + + for (i = 0; i < nr; i++) + f2fs_cache_wait_writeback(entries[i]); + f2fs_cache_gang_release(entries, nr); + goto next; +} + +static void f2fs_do_truncate_cache(struct f2fs_cached_block *entry, + bool drop_dirty) +{ + struct f2fs_cached_block_list *cache = entry->cache; + unsigned long flags; + + if (!drop_dirty && + (f2fs_cache_test_dirty(entry) || + f2fs_cache_test_writeback(entry))) + return; + + f2fs_cache_wait_writeback(entry); + f2fs_drop_cache_dirty(entry); + + spin_lock(&cache->list_lock); + spin_lock_irqsave(&cache->tree_lock, flags); + + f2fs_bug_on(cache->sbi, !entry->cache); + if (!radix_tree_delete(&cache->root, entry->index)) + f2fs_bug_on(cache->sbi, !entry->cache); + + entry->cache = NULL; + cache->num_entries--; + + atomic_dec(&entry->refcount); + f2fs_bug_on(cache->sbi, !f2fs_cache_refcount(entry)); + + f2fs_bug_on(cache->sbi, list_empty(&entry->list)); + list_del_init(&entry->list); + + spin_unlock_irqrestore(&cache->tree_lock, flags); + spin_unlock(&cache->list_lock); +} + +static void f2fs_truncate_cache(struct f2fs_cached_block *entry, + bool drop_dirty) +{ + f2fs_lock_cache(entry); + if (entry->cache) + f2fs_do_truncate_cache(entry, drop_dirty); + f2fs_unlock_cache(entry); +} + +static void f2fs_drop_cache(struct f2fs_cached_block_list *cache, + block_t blkaddr, bool drop_dirty) +{ + struct f2fs_cached_block *entry; + + entry = f2fs_find_cache(cache, blkaddr); + if (IS_ERR(entry)) + return; + + f2fs_truncate_cache(entry, drop_dirty); + f2fs_put_cache(entry, false); +} + +void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache, + unsigned long start, unsigned long len, bool drop_dirty) +{ + unsigned long index = start; + unsigned long end = (ULONG_MAX - start < len) ? + ULONG_MAX : (start + len); + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; + unsigned long flags; + int nr, i; + + if (len == 1) + return f2fs_drop_cache(cache, index, drop_dirty); + +next: + spin_lock_irqsave(&cache->tree_lock, flags); + nr = radix_tree_gang_lookup(&cache->root, (void **)entries, index, + min((unsigned long)F2FS_ONSTACK_CACHES, end - index)); + if (!nr) + goto out_unlock; + + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; + + if (entry->index >= end) { + nr = i; + break; + } + f2fs_cache_get(entry); + } +out_unlock: + spin_unlock_irqrestore(&cache->tree_lock, flags); + if (!nr) + return; + + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; + + index = entry->index + 1; + + f2fs_truncate_cache(entry, drop_dirty); + } + f2fs_cache_gang_release(entries, nr); + + if (index < end) + goto next; +} + +int f2fs_init_cache(struct f2fs_sb_info *sbi, + struct f2fs_cached_block_list *cache, + enum f2fs_cache_type type) +{ + cache->sbi = sbi; + cache->type = type; + INIT_RADIX_TREE(&cache->root, GFP_ATOMIC); + spin_lock_init(&cache->tree_lock); + spin_lock_init(&cache->list_lock); + INIT_LIST_HEAD(&cache->lru_list); + cache->num_entries = 0; + + return 0; +} + +void f2fs_destroy_cache(struct f2fs_cached_block_list *cache) +{ + struct list_head *head = &cache->lru_list; + struct f2fs_cached_block *entry; + unsigned long flags; + + f2fs_cache_wait_on_all_writeback(cache); +next: + spin_lock(&cache->list_lock); + if (list_empty(head)) { + spin_unlock(&cache->list_lock); + return; + } + entry = list_first_entry(head, struct f2fs_cached_block, list); + + spin_lock_irqsave(&cache->tree_lock, flags); + radix_tree_delete(&cache->root, entry->index); + cache->num_entries--; + list_del_init(&entry->list); + spin_unlock_irqrestore(&cache->tree_lock, flags); + + spin_unlock(&cache->list_lock); + + /* wait on read cache IO */ + f2fs_lock_cache(entry); + /* wait on write cache IO */ + f2fs_cache_wait_writeback(entry); + f2fs_bug_on(cache->sbi, f2fs_cache_test_dirty(entry)); + f2fs_bug_on(cache->sbi, f2fs_cache_test_writeback(entry)); + f2fs_bug_on(cache->sbi, !list_empty(&entry->list)); + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(entry) != 1); + f2fs_put_cache(entry, true); + goto next; +} diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h new file mode 100644 index 000000000000..686a974008ae --- /dev/null +++ b/fs/f2fs/cache.h @@ -0,0 +1,179 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2026 Google LLC + * Author: Chao Yu <chaseyu@google.com> + */ +#ifndef _LINUX_F2FS_CACHE_H +#define _LINUX_F2FS_CACHE_H + +#include <linux/pagemap.h> +#include <linux/mm.h> +#include <linux/list.h> +#include <linux/radix-tree.h> +#include <linux/spinlock.h> +#include <linux/wait.h> +#include <linux/types.h> + +struct f2fs_rwsem; +struct f2fs_io_info; +enum page_type; + +/* Represents a single cached block (meta, node or compress) */ +struct f2fs_cached_block { + struct list_head list; /* LRU list head */ + struct f2fs_cached_block_list *cache; /* parent cache list */ + union { + /* chain for merged BIO */ + struct f2fs_cached_block *next_entry; + nid_t ino; /* inode number for compress cache */ + }; + unsigned long index; /* key in radix tree, (meta/compress: pba, node: nid) */ + unsigned long state; /* cache entry state (e.g., Dirty, UpToDate) */ + void *data; /* blocksize-aligned memory (4KB or 16KB) */ + atomic_t refcount; /* reference count */ +}; + +struct f2fs_sb_info; + +enum f2fs_cache_type { + F2FS_META_CACHE, + F2FS_NODE_CACHE, +}; + +/* Main cache control structure (per sb_info) */ +struct f2fs_cached_block_list { + struct f2fs_sb_info *sbi; /* Pointer to f2fs_sb_info */ + struct radix_tree_root root; /* Radix tree for cache lookup */ + spinlock_t tree_lock; /* Lock for radix tree */ + struct list_head lru_list; /* Single global LRU list */ + spinlock_t list_lock; /* Lock for LRU list */ + enum f2fs_cache_type type; /* Cache type (Node or Meta) */ + unsigned long num_entries; /* Current number of entries */ +}; + +#define IS_META_CACHE(cache) (cache->type == F2FS_META_CACHE) + +/* Flags for f2fs_cached_block state */ +enum f2fs_cached_state { + F2FS_BLOCK_LOCKED, /* cache entry is locked */ + F2FS_BLOCK_UPTODATE, /* cache data is valid */ + F2FS_BLOCK_DIRTY, /* cache data is dirty, need to writeback the data */ + F2FS_BLOCK_WRITEBACK, /* cache data is writeback state */ + F2FS_BLOCK_INLINE_DATA, /* indicate inline data */ +}; + +enum { + __F2FS_CACHE_CREATE, /* create the cache if there is no cache entry */ + __F2FS_CACHE_LOCK, /* get and lock the cache entry */ + __F2FS_CACHE_NOFAIL, /* do not allow failure */ +}; + +enum f2fs_cache_request_flag { + F2FS_CACHE_CREATE = 1 << __F2FS_CACHE_CREATE, + F2FS_CACHE_LOCK = 1 << __F2FS_CACHE_LOCK, + F2FS_CACHE_NOFAIL = 1 << __F2FS_CACHE_NOFAIL, +}; + +#define F2FS_CACHE_LOCK_CREATE (F2FS_CACHE_LOCK | F2FS_CACHE_CREATE) + +#define F2FS_ONSTACK_CACHES (32) + +#define F2FS_CACHE_FLAG_TEST_FUNC(name, flagname) \ +static inline bool f2fs_cache_test_##name( \ + const struct f2fs_cached_block *entry) \ +{ \ + return test_bit(F2FS_BLOCK_##flagname, &entry->state); \ +} \ + +#define F2FS_CACHE_FLAG_SET_FUNC(name, flagname) \ +static inline void f2fs_cache_set_##name( \ + struct f2fs_cached_block *entry) \ +{ \ + set_bit(F2FS_BLOCK_##flagname, &entry->state); \ +} \ + +#define F2FS_CACHE_FLAG_CLEAR_FUNC(name, flagname) \ +static inline void f2fs_cache_clear_##name( \ + struct f2fs_cached_block *entry) \ +{ \ + clear_bit(F2FS_BLOCK_##flagname, &entry->state); \ +} \ + +#define F2FS_CACHE_FLAG_TEST_AND_SET_FUNC(name, flagname) \ +static inline bool f2fs_cache_test_and_set_##name( \ + struct f2fs_cached_block *entry) \ +{ \ + return test_and_set_bit(F2FS_BLOCK_##flagname, &entry->state); \ +} \ + +F2FS_CACHE_FLAG_TEST_FUNC(locked, LOCKED); +F2FS_CACHE_FLAG_SET_FUNC(locked, LOCKED); +F2FS_CACHE_FLAG_CLEAR_FUNC(locked, LOCKED); + +F2FS_CACHE_FLAG_TEST_FUNC(uptodate, UPTODATE); +F2FS_CACHE_FLAG_SET_FUNC(uptodate, UPTODATE); +F2FS_CACHE_FLAG_CLEAR_FUNC(uptodate, UPTODATE); + +F2FS_CACHE_FLAG_TEST_FUNC(dirty, DIRTY); +F2FS_CACHE_FLAG_SET_FUNC(dirty, DIRTY); +F2FS_CACHE_FLAG_CLEAR_FUNC(dirty, DIRTY); +F2FS_CACHE_FLAG_TEST_AND_SET_FUNC(dirty, DIRTY); + +F2FS_CACHE_FLAG_TEST_FUNC(writeback, WRITEBACK); +F2FS_CACHE_FLAG_SET_FUNC(writeback, WRITEBACK); +F2FS_CACHE_FLAG_CLEAR_FUNC(writeback, WRITEBACK); + +F2FS_CACHE_FLAG_TEST_FUNC(inline, INLINE_DATA); +F2FS_CACHE_FLAG_SET_FUNC(inline, INLINE_DATA); +F2FS_CACHE_FLAG_CLEAR_FUNC(inline, INLINE_DATA); + +static inline void *cache_address(const struct f2fs_cached_block *entry) +{ + return entry->data; +} + +#define CACHED_NODE(entry) ((struct f2fs_node *)(cache_address(entry))) + +static inline struct folio *cache_folio(const struct f2fs_cached_block *entry) +{ + return virt_to_folio(entry->data); +} + +int f2fs_init_cache(struct f2fs_sb_info *sbi, + struct f2fs_cached_block_list *cache, + enum f2fs_cache_type type); +void f2fs_destroy_cache(struct f2fs_cached_block_list *cache); +void f2fs_cache_get(struct f2fs_cached_block *entry); +struct f2fs_cached_block *f2fs_find_cache( + struct f2fs_cached_block_list *cache, + unsigned long index); +#define F2FS_CACHE_TAG_NONE 0 +#define F2FS_CACHE_TAG_DIRTY 1 +#define F2FS_CACHE_TAG_WRITEBACK 2 + +bool f2fs_trylock_cache(struct f2fs_cached_block *entry); +void f2fs_lock_cache(struct f2fs_cached_block *entry); +void f2fs_unlock_cache(struct f2fs_cached_block *entry); +bool f2fs_put_cache(struct f2fs_cached_block *entry, bool unlock); +bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry); +bool f2fs_clear_cache_dirty(struct f2fs_cached_block *entry); +void f2fs_drop_cache_dirty(struct f2fs_cached_block *entry); +void f2fs_force_clear_cache_dirty(struct f2fs_cached_block *entry); +void f2fs_start_cache_writeback(struct f2fs_cached_block *entry); +void f2fs_end_cache_writeback(struct f2fs_cached_block *entry); +unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache, + struct f2fs_cached_block **results, pgoff_t *first_index, + unsigned int max_items, int tag); +void f2fs_cache_gang_release(struct f2fs_cached_block **entries, + unsigned int nr_entries); +int f2fs_writeback_cache(struct f2fs_cached_block_list *cache, bool sync); +void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache); +void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, + enum page_type type); +void f2fs_cache_wait_writeback(struct f2fs_cached_block *entry); +struct f2fs_cached_block *f2fs_grab_cache(struct f2fs_cached_block_list *cache, + unsigned long index, int flags); +void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache, + unsigned long start, unsigned long len, bool drop_dirty); + +#endif /* _LINUX_F2FS_CACHE_H */ diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index 6ae0eb37d20f..09474790035b 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -41,11 +41,6 @@ struct f2fs_folio_state { unsigned int read_pages_pending; }; -struct f2fs_bio { - struct work_struct work; - struct bio bio; -}; - #define F2FS_BIO_POOL_SIZE NR_CURSEG_TYPE int __init f2fs_init_bioset(void) @@ -69,7 +64,6 @@ bool f2fs_is_cp_guaranteed(const struct folio *folio) return folio_test_f2fs_gcing(fscrypt_pagecache_folio(folio)); inode = mapping->host; - sbi = F2FS_I_SB(inode); if (inode->i_ino == F2FS_META_INO(sbi) || inode->i_ino == F2FS_NODE_INO(sbi) || @@ -437,6 +431,65 @@ static void f2fs_write_end_io(struct bio *bio) } } +static void f2fs_cache_read_end_io(struct bio *bio) +{ + struct f2fs_cached_block *entry = F2FS_BIO(bio)->entry; + struct f2fs_sb_info *sbi = entry->cache->sbi; + enum count_type io_type = IS_META_CACHE(entry->cache) ? + F2FS_RD_META : F2FS_RD_NODE; + struct f2fs_cached_block *next; + + iostat_update_and_unbind_ctx(bio); + + if (time_to_inject(sbi, FAULT_READ_IO)) + bio->bi_status = BLK_STS_IOERR; + + while (entry) { + next = entry->next_entry; + entry->next_entry = NULL; + + if (bio->bi_status == BLK_STS_OK) + f2fs_cache_set_uptodate(entry); + + dec_page_count(sbi, io_type); + + f2fs_unlock_cache(entry); + entry = next; + } + bio_put(bio); +} + +static void f2fs_cache_write_end_io(struct bio *bio) +{ + struct f2fs_cached_block *entry = F2FS_BIO(bio)->entry; + struct f2fs_sb_info *sbi = entry->cache->sbi; + struct f2fs_cached_block *next; + + iostat_update_and_unbind_ctx(bio); + + if (time_to_inject(sbi, FAULT_WRITE_IO)) + bio->bi_status = BLK_STS_IOERR; + + if (bio->bi_status != BLK_STS_OK) + f2fs_stop_checkpoint(sbi, true, + STOP_CP_REASON_WRITE_FAIL); + + while (entry) { + next = entry->next_entry; + entry->next_entry = NULL; + + dec_page_count(sbi, F2FS_WB_CP_DATA); + + if (!get_pages(sbi, F2FS_WB_CP_DATA) && + wq_has_sleeper(&sbi->cp_wait)) + wake_up(&sbi->cp_wait); + + f2fs_end_cache_writeback(entry); + entry = next; + } + bio_put(bio); +} + #ifdef CONFIG_BLK_DEV_ZONED static void f2fs_zone_write_end_io(struct bio *bio) { @@ -444,7 +497,10 @@ static void f2fs_zone_write_end_io(struct bio *bio) bio->bi_private = io->bi_private; complete(&io->zone_wait); - f2fs_write_end_io(bio); + if (io->fio.is_cache) + f2fs_cache_write_end_io(bio); + else + f2fs_write_end_io(bio); } #endif @@ -531,12 +587,21 @@ static struct bio *__bio_alloc(struct f2fs_io_info *fio, int npages) fio->op | fio->op_flags | f2fs_io_flags(fio), GFP_NOIO, &f2fs_bioset); bio->bi_iter.bi_sector = sector; + F2FS_BIO(bio)->entry = NULL; + bio->bi_private = NULL; if (is_read_io(fio->op)) { - bio->bi_end_io = f2fs_read_end_io; - bio->bi_private = NULL; + if (fio->is_cache) + bio->bi_end_io = f2fs_cache_read_end_io; + else + bio->bi_end_io = f2fs_read_end_io; } else { - bio->bi_end_io = f2fs_write_end_io; - bio->bi_private = sbi; + if (fio->is_cache) { + bio->bi_end_io = f2fs_cache_write_end_io; + } else { + bio->bi_end_io = f2fs_write_end_io; + bio->bi_private = sbi; + } + bio->bi_write_hint = f2fs_io_type_to_rw_hint(sbi, fio->type, fio->temp); bio->bi_write_stream = f2fs_io_type_to_write_stream(bdev, fio->type, @@ -559,7 +624,7 @@ static void f2fs_set_bio_crypt_ctx(struct bio *bio, const struct inode *inode, * The f2fs garbage collector sets ->encrypted_page when it wants to * read/write raw data without encryption. */ - if (!fio || !fio->encrypted_page) + if (!fio || (!fio->encrypted_page && !fio->is_cache)) fscrypt_set_bio_crypt_ctx(bio, inode, (loff_t)first_idx << inode->i_blkbits, gfp_mask); @@ -769,6 +834,53 @@ void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, __submit_merged_write_cond(sbi, NULL, folio, 0, type, true); } +static bool __has_merged_cache(struct bio *bio, + struct f2fs_cached_block *target) +{ + struct f2fs_cached_block *entry; + + if (!bio) + return false; + + entry = F2FS_BIO(bio)->entry; + + while (entry) { + if (target && entry == target) + return true; + entry = entry->next_entry; + } + return false; +} + +bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, + enum page_type type) +{ + struct f2fs_sb_info *sbi = entry->cache->sbi; + enum temp_type temp; + bool ret = false; + + for (temp = HOT; temp < NR_TEMP_TYPE; temp++) { + enum page_type btype = PAGE_TYPE_OF_BIO(type); + struct f2fs_bio_info *io = sbi->write_io[btype] + temp; + struct f2fs_lock_context lc; + bool merged; + + f2fs_down_read_trace(&io->io_rwsem, &lc); + merged = __has_merged_cache(io->bio, entry); + f2fs_up_read_trace(&io->io_rwsem, &lc); + + if (merged) { + __f2fs_submit_merged_write(sbi, type, temp); + ret = true; + } + + /* TODO: use HOT temp only for meta pages now. */ + if (type >= META) + break; + } + return ret; +} + void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi) { f2fs_submit_merged_write(sbi, DATA); @@ -832,6 +944,8 @@ static bool io_type_is_mergeable(struct f2fs_bio_info *io, if (io->fio.op != fio->op) return false; + if (io->fio.is_cache != fio->is_cache) + return false; return (io->fio.op_flags & mask) == (fio->op_flags & mask); } @@ -1063,6 +1177,34 @@ static bool is_end_zone_blkaddr(struct f2fs_sb_info *sbi, block_t blkaddr) f2fs_blkz_is_seq(sbi, devi, blkaddr) && (blkaddr % sbi->blocks_per_blkz == sbi->blocks_per_blkz - 1); } + +static void f2fs_wait_zone_io_completion(struct f2fs_sb_info *sbi, + struct f2fs_bio_info *io, enum page_type btype) +{ + if (f2fs_sb_has_blkzoned(sbi) && btype < META && io->zone_pending_bio) { + wait_for_completion_io(&io->zone_wait); + bio_put(io->zone_pending_bio); + io->zone_pending_bio = NULL; + io->bi_private = NULL; + } +} + +static void f2fs_submit_zone_io(struct f2fs_sb_info *sbi, + struct f2fs_io_info *fio, struct f2fs_bio_info *io, + enum page_type btype) +{ + if (f2fs_sb_has_blkzoned(sbi) && btype < META && + is_end_zone_blkaddr(sbi, fio->new_blkaddr)) { + bio_get(io->bio); + reinit_completion(&io->zone_wait); + io->bi_private = io->bio->bi_private; + io->bio->bi_private = io; + io->bio->bi_end_io = f2fs_zone_write_end_io; + io->zone_pending_bio = io->bio; + __submit_merged_bio(io); + } + +} #endif void f2fs_submit_page_write(struct f2fs_io_info *fio) @@ -1079,14 +1221,8 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio) f2fs_down_write_trace(&io->io_rwsem, &lc); next: #ifdef CONFIG_BLK_DEV_ZONED - if (f2fs_sb_has_blkzoned(sbi) && btype < META && io->zone_pending_bio) { - wait_for_completion_io(&io->zone_wait); - bio_put(io->zone_pending_bio); - io->zone_pending_bio = NULL; - io->bi_private = NULL; - } + f2fs_wait_zone_io_completion(sbi, io, btype); #endif - if (fio->in_list) { spin_lock(&io->io_lock); if (list_empty(&io->io_list)) { @@ -1141,16 +1277,109 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio) trace_f2fs_submit_folio_write(fio->folio, fio); #ifdef CONFIG_BLK_DEV_ZONED - if (f2fs_sb_has_blkzoned(sbi) && btype < META && - is_end_zone_blkaddr(sbi, fio->new_blkaddr)) { - bio_get(io->bio); - reinit_completion(&io->zone_wait); - io->bi_private = io->bio->bi_private; - io->bio->bi_private = io; - io->bio->bi_end_io = f2fs_zone_write_end_io; - io->zone_pending_bio = io->bio; + f2fs_submit_zone_io(sbi, fio, io, btype); +#endif + + if (fio->in_list) + goto next; +out: + if (is_sbi_flag_set(sbi, SBI_IS_SHUTDOWN) || + !f2fs_is_checkpoint_ready(sbi)) __submit_merged_bio(io); + f2fs_up_write_trace(&io->io_rwsem, &lc); +} + +static void f2fs_bio_add_cache(struct f2fs_io_info *fio, struct bio *bio) +{ + struct f2fs_bio *fbio = F2FS_BIO(bio); + struct f2fs_cached_block *head = fbio->entry; + struct f2fs_cached_block *new = fio->cache_entry; + + new->next_entry = head; + fbio->entry = new; +} + +int f2fs_submit_cache_read(struct f2fs_io_info *fio) +{ + struct f2fs_sb_info *sbi = fio->sbi; + struct f2fs_cached_block *entry = fio->cache_entry; + struct bio *bio; + enum count_type io_type = IS_META_CACHE(entry->cache) ? + F2FS_RD_META : F2FS_RD_NODE; + + if (!f2fs_is_valid_blkaddr(fio->sbi, fio->new_blkaddr, + fio->is_por ? META_POR : (__is_meta_io(fio) ? + META_GENERIC : DATA_GENERIC_ENHANCE))) + return -EFSCORRUPTED; + + bio = __bio_alloc(fio, 1); + + bio_add_virt_nofail(bio, cache_address(entry), sbi->blocksize); + f2fs_bio_add_cache(fio, bio); + inc_page_count(sbi, io_type); + + f2fs_submit_read_bio(sbi, bio, fio->type); + return 0; +} + +void f2fs_submit_cache_write(struct f2fs_io_info *fio) +{ + struct f2fs_sb_info *sbi = fio->sbi; + enum page_type btype = PAGE_TYPE_OF_BIO(fio->type); + struct f2fs_bio_info *io = sbi->write_io[btype] + fio->temp; + struct f2fs_lock_context lc; + struct folio *folio; + + f2fs_bug_on(sbi, is_read_io(fio->op)); + + f2fs_down_write_trace(&io->io_rwsem, &lc); +next: +#ifdef CONFIG_BLK_DEV_ZONED + f2fs_wait_zone_io_completion(sbi, io, btype); +#endif + if (fio->in_list) { + spin_lock(&io->io_lock); + if (list_empty(&io->io_list)) { + spin_unlock(&io->io_lock); + goto out; + } + fio = list_first_entry(&io->io_list, + struct f2fs_io_info, list); + list_del(&fio->list); + spin_unlock(&io->io_lock); } + + verify_fio_blkaddr(fio); + + fio->submitted = 1; + inc_page_count(sbi, F2FS_WB_CP_DATA); + + if (io->bio && + (!io_is_mergeable(sbi, io->bio, io, fio, io->last_block_in_bio, + fio->new_blkaddr))) + __submit_merged_bio(io); +alloc_new: + if (io->bio == NULL) { + io->bio = __bio_alloc(fio, BIO_MAX_VECS); + io->fio = *fio; + } + + folio = cache_folio(fio->cache_entry); + + if (!bio_add_folio(io->bio, folio, sbi->blocksize, + offset_in_folio(folio, cache_address(fio->cache_entry)))) { + f2fs_bug_on(sbi, !F2FS_BIO(io->bio)->entry); + + __submit_merged_bio(io); + goto alloc_new; + } + + f2fs_bio_add_cache(fio, io->bio); + + io->last_block_in_bio = fio->new_blkaddr; + +#ifdef CONFIG_BLK_DEV_ZONED + f2fs_submit_zone_io(sbi, fio, io, btype); #endif if (fio->in_list) goto next; diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 1b96d8718c5c..8413983ea9d5 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -221,6 +221,8 @@ struct f2fs_rwsem { #endif }; +#include "cache.h" + struct f2fs_mount_info { unsigned long long opt; block_t root_reserved_blocks; /* root reserved blocks */ @@ -1370,8 +1372,10 @@ struct f2fs_io_info { unsigned int is_por:1; /* indicate IO is from recovery or not */ unsigned int encrypted:1; /* indicate file is encrypted */ unsigned int meta_gc:1; /* require meta inode GC */ + unsigned int is_cache:1; /* indicate IO is from internal cache */ enum iostat_type io_type; /* io type */ struct writeback_control *io_wbc; /* writeback control */ + struct f2fs_cached_block *cache_entry; struct bio **bio; /* bio for ipu */ sector_t *last_block; /* last block number in bio */ }; @@ -1781,6 +1785,12 @@ struct f2fs_gc_kthread { unsigned int boost_gc_greedy; }; +struct f2fs_bio { + struct work_struct work; + struct f2fs_cached_block *entry; + struct bio bio; +}; + struct f2fs_sb_info { struct super_block *sb; /* pointer to VFS super block */ struct proc_dir_entry *s_proc; /* proc entry */ @@ -2323,6 +2333,16 @@ static inline bool is_node_folio(struct folio *folio) return folio->mapping == NODE_MAPPING(F2FS_F_SB(folio)); } +static inline struct f2fs_bio *F2FS_BIO(struct bio *bio) +{ + return container_of(bio, struct f2fs_bio, bio); +} + +static inline bool f2fs_is_cache_bio(struct bio *bio) +{ + return F2FS_BIO(bio)->entry != NULL; +} + static inline bool is_sbi_flag_set(struct f2fs_sb_info *sbi, unsigned int type) { return test_bit(type, &sbi->s_flag); @@ -4237,6 +4257,8 @@ void f2fs_submit_merged_write_cond(struct f2fs_sb_info *sbi, nid_t ino, enum page_type type); void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, struct folio *folio, enum page_type type); +bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, + enum page_type type); void f2fs_submit_merged_ipu_write(struct f2fs_sb_info *sbi, struct bio **bio, struct folio *folio); void f2fs_submit_all_merged_ipu_writes(struct f2fs_sb_info *sbi); @@ -4244,6 +4266,8 @@ void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi); int f2fs_submit_page_bio(struct f2fs_io_info *fio); int f2fs_merge_page_bio(struct f2fs_io_info *fio); void f2fs_submit_page_write(struct f2fs_io_info *fio); +int f2fs_submit_cache_read(struct f2fs_io_info *fio); +void f2fs_submit_cache_write(struct f2fs_io_info *fio); struct block_device *f2fs_target_device(struct f2fs_sb_info *sbi, block_t blk_addr, sector_t *sector); int f2fs_target_device_index(struct f2fs_sb_info *sbi, block_t blkaddr); diff --git a/fs/f2fs/iostat.h b/fs/f2fs/iostat.h index 2025225b5bed..61c6bc8e3119 100644 --- a/fs/f2fs/iostat.h +++ b/fs/f2fs/iostat.h @@ -60,6 +60,13 @@ static inline struct bio_post_read_ctx *get_post_read_ctx(struct bio *bio) return iostat_ctx->post_read_ctx; } +static inline void iostat_set_post_read_ctx(struct bio *bio, void *ctx) +{ + struct bio_iostat_ctx *iostat_ctx = bio->bi_private; + + iostat_ctx->post_read_ctx = ctx; +} + extern void iostat_update_and_unbind_ctx(struct bio *bio); extern void iostat_alloc_and_bind_ctx(struct f2fs_sb_info *sbi, struct bio *bio, struct bio_post_read_ctx *ctx); @@ -81,6 +88,10 @@ static inline struct bio_post_read_ctx *get_post_read_ctx(struct bio *bio) { return bio->bi_private; } +static inline void iostat_set_post_read_ctx(struct bio *bio, void *ctx) +{ + bio->bi_private = ctx; +} static inline int f2fs_init_iostat_processing(void) { return 0; } static inline void f2fs_destroy_iostat_processing(void) {} static inline int f2fs_init_iostat(struct f2fs_sb_info *sbi) { return 0; } -- 2.49.0 ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [f2fs-dev] [PATCH v1 01/12] f2fs: cache: implement metadata cache @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-kernel, linux-f2fs-devel This patch introduces the core metadata block caching infrastructure to manage f2fs metadata independently of the page cache. It implements: - core cache APIs: get, create, put, drop, backed by a radix tree and a single global LRU list. - support multiple status of cached block: LOCKED, UPTODATE, DIRTY, WRITEBACK, INLINE. - internal bio based read/write helpers with adjacent block vector merging. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/Makefile | 2 +- fs/f2fs/cache.c | 531 +++++++++++++++++++++++++++++++++++++++++++++++ fs/f2fs/cache.h | 179 ++++++++++++++++ fs/f2fs/data.c | 283 ++++++++++++++++++++++--- fs/f2fs/f2fs.h | 24 +++ fs/f2fs/iostat.h | 11 + 6 files changed, 1002 insertions(+), 28 deletions(-) create mode 100644 fs/f2fs/cache.c create mode 100644 fs/f2fs/cache.h diff --git a/fs/f2fs/Makefile b/fs/f2fs/Makefile index 8a7322d229e4..fbf49c30b066 100644 --- a/fs/f2fs/Makefile +++ b/fs/f2fs/Makefile @@ -3,7 +3,7 @@ obj-$(CONFIG_F2FS_FS) += f2fs.o f2fs-y := dir.o file.o inode.o namei.o hash.o super.o inline.o f2fs-y += checkpoint.o gc.o data.o node.o segment.o recovery.o -f2fs-y += shrinker.o extent_cache.o sysfs.o +f2fs-y += shrinker.o extent_cache.o sysfs.o cache.o f2fs-$(CONFIG_F2FS_STAT_FS) += debug.o f2fs-$(CONFIG_F2FS_FS_XATTR) += xattr.o f2fs-$(CONFIG_F2FS_FS_POSIX_ACL) += acl.o diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c new file mode 100644 index 000000000000..08bc658166f7 --- /dev/null +++ b/fs/f2fs/cache.c @@ -0,0 +1,531 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 Google LLC + * Author: Chao Yu <chaseyu@google.com> + */ +#include "linux/spinlock.h" +#include <linux/fs.h> +#include <linux/f2fs_fs.h> +#include <linux/radix-tree.h> +#include <linux/slab.h> +#include <linux/list.h> +#include <linux/pagemap.h> +#include <linux/kthread.h> +#include <linux/freezer.h> +#include <linux/delay.h> +#include "f2fs.h" +#include "cache.h" +#include "node.h" +#include "segment.h" + +void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, + enum page_type type) +{ + /* in case the entry was truncated or on-going shrink */ + if (!entry->cache) + return; + + if (!f2fs_cache_test_writeback(entry)) + return; + + /* submit cached bio */ + f2fs_submit_merged_write_cache(entry, type); + + wait_on_bit_io(&entry->state, F2FS_BLOCK_WRITEBACK, + TASK_UNINTERRUPTIBLE); +} + +void f2fs_cache_wait_writeback(struct f2fs_cached_block *entry) +{ + /* in case the entry was truncated or on-going shrink */ + if (!entry->cache) + return; + + f2fs_cache_wait_writeback_cond(entry, + IS_META_CACHE(entry->cache) ? META : NODE); +} + +static void f2fs_cache_update_tag(struct f2fs_cached_block *entry, unsigned int src, + unsigned int dst) +{ + struct f2fs_cached_block_list *cache = entry->cache; + unsigned long flags; + + spin_lock_irqsave(&cache->tree_lock, flags); + if (src) + radix_tree_tag_clear(&cache->root, entry->index, src); + if (dst) + radix_tree_tag_set(&cache->root, entry->index, dst); + spin_unlock_irqrestore(&cache->tree_lock, flags); +} + +bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry) +{ + struct f2fs_cached_block_list *cache = entry->cache; + + if (!f2fs_cache_test_uptodate(entry)) + f2fs_cache_set_uptodate(entry); + +#ifdef CONFIG_F2FS_CHECK_FS + if (cache->type == F2FS_NODE_CACHE && IS_INODE(cache_folio(entry))) + f2fs_inode_chksum_set(cache->sbi, cache_folio(entry)); +#endif + + if (f2fs_cache_test_dirty(entry)) + return false; + + if (!f2fs_cache_test_and_set_dirty(entry)) { + enum count_type type = IS_META_CACHE(cache) ? + F2FS_DIRTY_META : F2FS_DIRTY_NODES; + + f2fs_cache_update_tag(entry, 0, F2FS_CACHE_TAG_DIRTY); + inc_page_count(cache->sbi, type); + return true; + } + + return false; +} + +bool f2fs_clear_cache_dirty(struct f2fs_cached_block *entry) +{ + if (!f2fs_cache_test_dirty(entry)) + return false; + + f2fs_cache_clear_dirty(entry); + return true; +} + +static void __f2fs_drop_cache_dirty(struct f2fs_cached_block *entry, bool force) +{ + + struct f2fs_cached_block_list *cache = entry->cache; + enum count_type type = IS_META_CACHE(cache) ? + F2FS_DIRTY_META : F2FS_DIRTY_NODES; + + f2fs_cache_clear_uptodate(entry); + + if (!force && !f2fs_clear_cache_dirty(entry)) + return; + + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_DIRTY, 0); + dec_page_count(cache->sbi, type); +} + +void f2fs_drop_cache_dirty(struct f2fs_cached_block *entry) +{ + __f2fs_drop_cache_dirty(entry, false); +} + +void f2fs_force_clear_cache_dirty(struct f2fs_cached_block *entry) +{ + __f2fs_drop_cache_dirty(entry, true); +} + +void f2fs_start_cache_writeback(struct f2fs_cached_block *entry) +{ + f2fs_cache_set_writeback(entry); + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_DIRTY, + F2FS_CACHE_TAG_WRITEBACK); +} + +void f2fs_end_cache_writeback(struct f2fs_cached_block *entry) +{ + /* + * should call f2fs_cache_update_tag() before clearing writeback bit, + * in case f2fs_truncate_cache() set entry->cache to NULL. + */ + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_WRITEBACK, 0); + clear_and_wake_up_bit(F2FS_BLOCK_WRITEBACK, &entry->state); +} + +static int f2fs_cache_refcount(struct f2fs_cached_block *entry) +{ + return atomic_read(&entry->refcount); +} + +static void __f2fs_free_cache(struct f2fs_cached_block *entry) +{ + kfree(entry->data); + kfree(entry); +} + +static void f2fs_free_cache(struct f2fs_cached_block *entry) +{ + WARN_ON_ONCE(!list_empty(&entry->list)); + WARN_ON_ONCE(f2fs_cache_refcount(entry)); + __f2fs_free_cache(entry); +} + +void f2fs_cache_get(struct f2fs_cached_block *entry) +{ + atomic_inc(&entry->refcount); +} + +static bool f2fs_cache_put(struct f2fs_cached_block *entry) +{ + WARN_ON_ONCE(!f2fs_cache_refcount(entry)); + if (atomic_dec_and_test(&entry->refcount)) { + f2fs_free_cache(entry); + return true; + } + return false; +} + +static struct f2fs_cached_block *f2fs_create_cache( + struct f2fs_cached_block_list *cache, + unsigned long index, bool nofail) +{ + struct f2fs_cached_block *entry; + unsigned int flags = GFP_NOFS; + + if (nofail) + flags |= __GFP_NOFAIL; + + entry = kzalloc_obj(*entry, flags); + if (!entry) + return ERR_PTR(-ENOMEM); + + entry->data = kzalloc(cache->sbi->blocksize, flags); + if (!entry->data) { + kfree(entry); + return ERR_PTR(-ENOMEM); + } + + entry->index = index; + + atomic_set(&entry->refcount, 0); + entry->next_entry = NULL; + INIT_LIST_HEAD(&entry->list); + + entry->cache = cache; + + return entry; +} + +static struct f2fs_cached_block *f2fs_insert_cache( + struct f2fs_cached_block_list *cache, + unsigned long index, + struct f2fs_cached_block *new) +{ + struct f2fs_cached_block *e; + int ret; + unsigned long flags; + + ret = radix_tree_preload(GFP_NOFS | __GFP_NOFAIL); + f2fs_bug_on(cache->sbi, ret); + + spin_lock(&cache->list_lock); + spin_lock_irqsave(&cache->tree_lock, flags); + e = radix_tree_lookup(&cache->root, index); + if (!e) { + e = new; + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(e)); + + ret = radix_tree_insert(&cache->root, index, e); + f2fs_bug_on(cache->sbi, ret); + + /* radix tree referenced cache entry */ + f2fs_cache_get(e); + f2fs_bug_on(cache->sbi, !list_empty(&e->list)); + list_add_tail(&e->list, &cache->lru_list); + cache->num_entries++; + } + f2fs_cache_get(e); + spin_unlock_irqrestore(&cache->tree_lock, flags); + spin_unlock(&cache->list_lock); + radix_tree_preload_end(); + + if (new != e) { + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(new)); + __f2fs_free_cache(new); + } + + return e; +} + +struct f2fs_cached_block *f2fs_find_cache( + struct f2fs_cached_block_list *cache, + unsigned long index) +{ + struct f2fs_cached_block *entry; + unsigned long flags; + + spin_lock(&cache->list_lock); + spin_lock_irqsave(&cache->tree_lock, flags); + entry = radix_tree_lookup(&cache->root, index); + if (entry) { + f2fs_bug_on(cache->sbi, !f2fs_cache_refcount(entry)); + f2fs_cache_get(entry); + if (!list_empty(&entry->list)) + list_move_tail(&entry->list, &cache->lru_list); + + } else { + entry = ERR_PTR(-ENOENT); + } + spin_unlock_irqrestore(&cache->tree_lock, flags); + spin_unlock(&cache->list_lock); + + return entry; +} + +struct f2fs_cached_block *f2fs_grab_cache( + struct f2fs_cached_block_list *cache, + unsigned long index, int flags) + +{ + struct f2fs_cached_block *entry, *new; + bool create = flags & F2FS_CACHE_CREATE; + bool nofail = flags & F2FS_CACHE_NOFAIL; + bool lock = flags & F2FS_CACHE_LOCK; + +repeat: + entry = f2fs_find_cache(cache, index); + if (!IS_ERR(entry)) + goto found; + + if (!create) + return ERR_PTR(-ENOENT); + + new = f2fs_create_cache(cache, index, nofail); + if (IS_ERR(new)) + return new; + + entry = f2fs_insert_cache(cache, index, new); +found: + if (lock) { + f2fs_lock_cache(entry); + /* has been truncated */ + if (entry->cache != cache) { + f2fs_put_cache(entry, true); + goto repeat; + } + } + return entry; +} + +bool f2fs_trylock_cache(struct f2fs_cached_block *entry) +{ + return !test_and_set_bit(F2FS_BLOCK_LOCKED, &entry->state); +} + +void f2fs_lock_cache(struct f2fs_cached_block *entry) +{ + wait_on_bit_lock(&entry->state, F2FS_BLOCK_LOCKED, + TASK_UNINTERRUPTIBLE); +} + +void f2fs_unlock_cache(struct f2fs_cached_block *entry) +{ + clear_and_wake_up_bit(F2FS_BLOCK_LOCKED, &entry->state); +} + +bool f2fs_put_cache(struct f2fs_cached_block *entry, bool unlock) +{ + if (IS_ERR_OR_NULL(entry)) + return false; + if (unlock) + f2fs_unlock_cache(entry); + return f2fs_cache_put(entry); +} + +unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache, + struct f2fs_cached_block **entries, + pgoff_t *first_index, unsigned int max_nr, + int tag) +{ + unsigned long flags; + int nr, i; + + spin_lock_irqsave(&cache->tree_lock, flags); + nr = radix_tree_gang_lookup_tag(&cache->root, (void **)entries, + *first_index, max_nr, tag); + if (!nr) + goto out; + + for (i = 0; i < nr; i++) + f2fs_cache_get(entries[i]); + *first_index = entries[nr - 1]->index + 1; +out: + spin_unlock_irqrestore(&cache->tree_lock, flags); + return nr; +} + +void f2fs_cache_gang_release(struct f2fs_cached_block **entries, + unsigned int nr_entries) +{ + int i; + + for (i = 0; i < nr_entries; i++) + f2fs_put_cache(entries[i], false); +} + +void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache) +{ + unsigned long index = 0; + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; + int nr, i; + +next: + nr = f2fs_cache_gang_lookup_tag(cache, entries, &index, + F2FS_ONSTACK_CACHES, F2FS_CACHE_TAG_WRITEBACK); + if (!nr) + return; + + for (i = 0; i < nr; i++) + f2fs_cache_wait_writeback(entries[i]); + f2fs_cache_gang_release(entries, nr); + goto next; +} + +static void f2fs_do_truncate_cache(struct f2fs_cached_block *entry, + bool drop_dirty) +{ + struct f2fs_cached_block_list *cache = entry->cache; + unsigned long flags; + + if (!drop_dirty && + (f2fs_cache_test_dirty(entry) || + f2fs_cache_test_writeback(entry))) + return; + + f2fs_cache_wait_writeback(entry); + f2fs_drop_cache_dirty(entry); + + spin_lock(&cache->list_lock); + spin_lock_irqsave(&cache->tree_lock, flags); + + f2fs_bug_on(cache->sbi, !entry->cache); + if (!radix_tree_delete(&cache->root, entry->index)) + f2fs_bug_on(cache->sbi, !entry->cache); + + entry->cache = NULL; + cache->num_entries--; + + atomic_dec(&entry->refcount); + f2fs_bug_on(cache->sbi, !f2fs_cache_refcount(entry)); + + f2fs_bug_on(cache->sbi, list_empty(&entry->list)); + list_del_init(&entry->list); + + spin_unlock_irqrestore(&cache->tree_lock, flags); + spin_unlock(&cache->list_lock); +} + +static void f2fs_truncate_cache(struct f2fs_cached_block *entry, + bool drop_dirty) +{ + f2fs_lock_cache(entry); + if (entry->cache) + f2fs_do_truncate_cache(entry, drop_dirty); + f2fs_unlock_cache(entry); +} + +static void f2fs_drop_cache(struct f2fs_cached_block_list *cache, + block_t blkaddr, bool drop_dirty) +{ + struct f2fs_cached_block *entry; + + entry = f2fs_find_cache(cache, blkaddr); + if (IS_ERR(entry)) + return; + + f2fs_truncate_cache(entry, drop_dirty); + f2fs_put_cache(entry, false); +} + +void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache, + unsigned long start, unsigned long len, bool drop_dirty) +{ + unsigned long index = start; + unsigned long end = (ULONG_MAX - start < len) ? + ULONG_MAX : (start + len); + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; + unsigned long flags; + int nr, i; + + if (len == 1) + return f2fs_drop_cache(cache, index, drop_dirty); + +next: + spin_lock_irqsave(&cache->tree_lock, flags); + nr = radix_tree_gang_lookup(&cache->root, (void **)entries, index, + min((unsigned long)F2FS_ONSTACK_CACHES, end - index)); + if (!nr) + goto out_unlock; + + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; + + if (entry->index >= end) { + nr = i; + break; + } + f2fs_cache_get(entry); + } +out_unlock: + spin_unlock_irqrestore(&cache->tree_lock, flags); + if (!nr) + return; + + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; + + index = entry->index + 1; + + f2fs_truncate_cache(entry, drop_dirty); + } + f2fs_cache_gang_release(entries, nr); + + if (index < end) + goto next; +} + +int f2fs_init_cache(struct f2fs_sb_info *sbi, + struct f2fs_cached_block_list *cache, + enum f2fs_cache_type type) +{ + cache->sbi = sbi; + cache->type = type; + INIT_RADIX_TREE(&cache->root, GFP_ATOMIC); + spin_lock_init(&cache->tree_lock); + spin_lock_init(&cache->list_lock); + INIT_LIST_HEAD(&cache->lru_list); + cache->num_entries = 0; + + return 0; +} + +void f2fs_destroy_cache(struct f2fs_cached_block_list *cache) +{ + struct list_head *head = &cache->lru_list; + struct f2fs_cached_block *entry; + unsigned long flags; + + f2fs_cache_wait_on_all_writeback(cache); +next: + spin_lock(&cache->list_lock); + if (list_empty(head)) { + spin_unlock(&cache->list_lock); + return; + } + entry = list_first_entry(head, struct f2fs_cached_block, list); + + spin_lock_irqsave(&cache->tree_lock, flags); + radix_tree_delete(&cache->root, entry->index); + cache->num_entries--; + list_del_init(&entry->list); + spin_unlock_irqrestore(&cache->tree_lock, flags); + + spin_unlock(&cache->list_lock); + + /* wait on read cache IO */ + f2fs_lock_cache(entry); + /* wait on write cache IO */ + f2fs_cache_wait_writeback(entry); + f2fs_bug_on(cache->sbi, f2fs_cache_test_dirty(entry)); + f2fs_bug_on(cache->sbi, f2fs_cache_test_writeback(entry)); + f2fs_bug_on(cache->sbi, !list_empty(&entry->list)); + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(entry) != 1); + f2fs_put_cache(entry, true); + goto next; +} diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h new file mode 100644 index 000000000000..686a974008ae --- /dev/null +++ b/fs/f2fs/cache.h @@ -0,0 +1,179 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2026 Google LLC + * Author: Chao Yu <chaseyu@google.com> + */ +#ifndef _LINUX_F2FS_CACHE_H +#define _LINUX_F2FS_CACHE_H + +#include <linux/pagemap.h> +#include <linux/mm.h> +#include <linux/list.h> +#include <linux/radix-tree.h> +#include <linux/spinlock.h> +#include <linux/wait.h> +#include <linux/types.h> + +struct f2fs_rwsem; +struct f2fs_io_info; +enum page_type; + +/* Represents a single cached block (meta, node or compress) */ +struct f2fs_cached_block { + struct list_head list; /* LRU list head */ + struct f2fs_cached_block_list *cache; /* parent cache list */ + union { + /* chain for merged BIO */ + struct f2fs_cached_block *next_entry; + nid_t ino; /* inode number for compress cache */ + }; + unsigned long index; /* key in radix tree, (meta/compress: pba, node: nid) */ + unsigned long state; /* cache entry state (e.g., Dirty, UpToDate) */ + void *data; /* blocksize-aligned memory (4KB or 16KB) */ + atomic_t refcount; /* reference count */ +}; + +struct f2fs_sb_info; + +enum f2fs_cache_type { + F2FS_META_CACHE, + F2FS_NODE_CACHE, +}; + +/* Main cache control structure (per sb_info) */ +struct f2fs_cached_block_list { + struct f2fs_sb_info *sbi; /* Pointer to f2fs_sb_info */ + struct radix_tree_root root; /* Radix tree for cache lookup */ + spinlock_t tree_lock; /* Lock for radix tree */ + struct list_head lru_list; /* Single global LRU list */ + spinlock_t list_lock; /* Lock for LRU list */ + enum f2fs_cache_type type; /* Cache type (Node or Meta) */ + unsigned long num_entries; /* Current number of entries */ +}; + +#define IS_META_CACHE(cache) (cache->type == F2FS_META_CACHE) + +/* Flags for f2fs_cached_block state */ +enum f2fs_cached_state { + F2FS_BLOCK_LOCKED, /* cache entry is locked */ + F2FS_BLOCK_UPTODATE, /* cache data is valid */ + F2FS_BLOCK_DIRTY, /* cache data is dirty, need to writeback the data */ + F2FS_BLOCK_WRITEBACK, /* cache data is writeback state */ + F2FS_BLOCK_INLINE_DATA, /* indicate inline data */ +}; + +enum { + __F2FS_CACHE_CREATE, /* create the cache if there is no cache entry */ + __F2FS_CACHE_LOCK, /* get and lock the cache entry */ + __F2FS_CACHE_NOFAIL, /* do not allow failure */ +}; + +enum f2fs_cache_request_flag { + F2FS_CACHE_CREATE = 1 << __F2FS_CACHE_CREATE, + F2FS_CACHE_LOCK = 1 << __F2FS_CACHE_LOCK, + F2FS_CACHE_NOFAIL = 1 << __F2FS_CACHE_NOFAIL, +}; + +#define F2FS_CACHE_LOCK_CREATE (F2FS_CACHE_LOCK | F2FS_CACHE_CREATE) + +#define F2FS_ONSTACK_CACHES (32) + +#define F2FS_CACHE_FLAG_TEST_FUNC(name, flagname) \ +static inline bool f2fs_cache_test_##name( \ + const struct f2fs_cached_block *entry) \ +{ \ + return test_bit(F2FS_BLOCK_##flagname, &entry->state); \ +} \ + +#define F2FS_CACHE_FLAG_SET_FUNC(name, flagname) \ +static inline void f2fs_cache_set_##name( \ + struct f2fs_cached_block *entry) \ +{ \ + set_bit(F2FS_BLOCK_##flagname, &entry->state); \ +} \ + +#define F2FS_CACHE_FLAG_CLEAR_FUNC(name, flagname) \ +static inline void f2fs_cache_clear_##name( \ + struct f2fs_cached_block *entry) \ +{ \ + clear_bit(F2FS_BLOCK_##flagname, &entry->state); \ +} \ + +#define F2FS_CACHE_FLAG_TEST_AND_SET_FUNC(name, flagname) \ +static inline bool f2fs_cache_test_and_set_##name( \ + struct f2fs_cached_block *entry) \ +{ \ + return test_and_set_bit(F2FS_BLOCK_##flagname, &entry->state); \ +} \ + +F2FS_CACHE_FLAG_TEST_FUNC(locked, LOCKED); +F2FS_CACHE_FLAG_SET_FUNC(locked, LOCKED); +F2FS_CACHE_FLAG_CLEAR_FUNC(locked, LOCKED); + +F2FS_CACHE_FLAG_TEST_FUNC(uptodate, UPTODATE); +F2FS_CACHE_FLAG_SET_FUNC(uptodate, UPTODATE); +F2FS_CACHE_FLAG_CLEAR_FUNC(uptodate, UPTODATE); + +F2FS_CACHE_FLAG_TEST_FUNC(dirty, DIRTY); +F2FS_CACHE_FLAG_SET_FUNC(dirty, DIRTY); +F2FS_CACHE_FLAG_CLEAR_FUNC(dirty, DIRTY); +F2FS_CACHE_FLAG_TEST_AND_SET_FUNC(dirty, DIRTY); + +F2FS_CACHE_FLAG_TEST_FUNC(writeback, WRITEBACK); +F2FS_CACHE_FLAG_SET_FUNC(writeback, WRITEBACK); +F2FS_CACHE_FLAG_CLEAR_FUNC(writeback, WRITEBACK); + +F2FS_CACHE_FLAG_TEST_FUNC(inline, INLINE_DATA); +F2FS_CACHE_FLAG_SET_FUNC(inline, INLINE_DATA); +F2FS_CACHE_FLAG_CLEAR_FUNC(inline, INLINE_DATA); + +static inline void *cache_address(const struct f2fs_cached_block *entry) +{ + return entry->data; +} + +#define CACHED_NODE(entry) ((struct f2fs_node *)(cache_address(entry))) + +static inline struct folio *cache_folio(const struct f2fs_cached_block *entry) +{ + return virt_to_folio(entry->data); +} + +int f2fs_init_cache(struct f2fs_sb_info *sbi, + struct f2fs_cached_block_list *cache, + enum f2fs_cache_type type); +void f2fs_destroy_cache(struct f2fs_cached_block_list *cache); +void f2fs_cache_get(struct f2fs_cached_block *entry); +struct f2fs_cached_block *f2fs_find_cache( + struct f2fs_cached_block_list *cache, + unsigned long index); +#define F2FS_CACHE_TAG_NONE 0 +#define F2FS_CACHE_TAG_DIRTY 1 +#define F2FS_CACHE_TAG_WRITEBACK 2 + +bool f2fs_trylock_cache(struct f2fs_cached_block *entry); +void f2fs_lock_cache(struct f2fs_cached_block *entry); +void f2fs_unlock_cache(struct f2fs_cached_block *entry); +bool f2fs_put_cache(struct f2fs_cached_block *entry, bool unlock); +bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry); +bool f2fs_clear_cache_dirty(struct f2fs_cached_block *entry); +void f2fs_drop_cache_dirty(struct f2fs_cached_block *entry); +void f2fs_force_clear_cache_dirty(struct f2fs_cached_block *entry); +void f2fs_start_cache_writeback(struct f2fs_cached_block *entry); +void f2fs_end_cache_writeback(struct f2fs_cached_block *entry); +unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache, + struct f2fs_cached_block **results, pgoff_t *first_index, + unsigned int max_items, int tag); +void f2fs_cache_gang_release(struct f2fs_cached_block **entries, + unsigned int nr_entries); +int f2fs_writeback_cache(struct f2fs_cached_block_list *cache, bool sync); +void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache); +void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, + enum page_type type); +void f2fs_cache_wait_writeback(struct f2fs_cached_block *entry); +struct f2fs_cached_block *f2fs_grab_cache(struct f2fs_cached_block_list *cache, + unsigned long index, int flags); +void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache, + unsigned long start, unsigned long len, bool drop_dirty); + +#endif /* _LINUX_F2FS_CACHE_H */ diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index 6ae0eb37d20f..09474790035b 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -41,11 +41,6 @@ struct f2fs_folio_state { unsigned int read_pages_pending; }; -struct f2fs_bio { - struct work_struct work; - struct bio bio; -}; - #define F2FS_BIO_POOL_SIZE NR_CURSEG_TYPE int __init f2fs_init_bioset(void) @@ -69,7 +64,6 @@ bool f2fs_is_cp_guaranteed(const struct folio *folio) return folio_test_f2fs_gcing(fscrypt_pagecache_folio(folio)); inode = mapping->host; - sbi = F2FS_I_SB(inode); if (inode->i_ino == F2FS_META_INO(sbi) || inode->i_ino == F2FS_NODE_INO(sbi) || @@ -437,6 +431,65 @@ static void f2fs_write_end_io(struct bio *bio) } } +static void f2fs_cache_read_end_io(struct bio *bio) +{ + struct f2fs_cached_block *entry = F2FS_BIO(bio)->entry; + struct f2fs_sb_info *sbi = entry->cache->sbi; + enum count_type io_type = IS_META_CACHE(entry->cache) ? + F2FS_RD_META : F2FS_RD_NODE; + struct f2fs_cached_block *next; + + iostat_update_and_unbind_ctx(bio); + + if (time_to_inject(sbi, FAULT_READ_IO)) + bio->bi_status = BLK_STS_IOERR; + + while (entry) { + next = entry->next_entry; + entry->next_entry = NULL; + + if (bio->bi_status == BLK_STS_OK) + f2fs_cache_set_uptodate(entry); + + dec_page_count(sbi, io_type); + + f2fs_unlock_cache(entry); + entry = next; + } + bio_put(bio); +} + +static void f2fs_cache_write_end_io(struct bio *bio) +{ + struct f2fs_cached_block *entry = F2FS_BIO(bio)->entry; + struct f2fs_sb_info *sbi = entry->cache->sbi; + struct f2fs_cached_block *next; + + iostat_update_and_unbind_ctx(bio); + + if (time_to_inject(sbi, FAULT_WRITE_IO)) + bio->bi_status = BLK_STS_IOERR; + + if (bio->bi_status != BLK_STS_OK) + f2fs_stop_checkpoint(sbi, true, + STOP_CP_REASON_WRITE_FAIL); + + while (entry) { + next = entry->next_entry; + entry->next_entry = NULL; + + dec_page_count(sbi, F2FS_WB_CP_DATA); + + if (!get_pages(sbi, F2FS_WB_CP_DATA) && + wq_has_sleeper(&sbi->cp_wait)) + wake_up(&sbi->cp_wait); + + f2fs_end_cache_writeback(entry); + entry = next; + } + bio_put(bio); +} + #ifdef CONFIG_BLK_DEV_ZONED static void f2fs_zone_write_end_io(struct bio *bio) { @@ -444,7 +497,10 @@ static void f2fs_zone_write_end_io(struct bio *bio) bio->bi_private = io->bi_private; complete(&io->zone_wait); - f2fs_write_end_io(bio); + if (io->fio.is_cache) + f2fs_cache_write_end_io(bio); + else + f2fs_write_end_io(bio); } #endif @@ -531,12 +587,21 @@ static struct bio *__bio_alloc(struct f2fs_io_info *fio, int npages) fio->op | fio->op_flags | f2fs_io_flags(fio), GFP_NOIO, &f2fs_bioset); bio->bi_iter.bi_sector = sector; + F2FS_BIO(bio)->entry = NULL; + bio->bi_private = NULL; if (is_read_io(fio->op)) { - bio->bi_end_io = f2fs_read_end_io; - bio->bi_private = NULL; + if (fio->is_cache) + bio->bi_end_io = f2fs_cache_read_end_io; + else + bio->bi_end_io = f2fs_read_end_io; } else { - bio->bi_end_io = f2fs_write_end_io; - bio->bi_private = sbi; + if (fio->is_cache) { + bio->bi_end_io = f2fs_cache_write_end_io; + } else { + bio->bi_end_io = f2fs_write_end_io; + bio->bi_private = sbi; + } + bio->bi_write_hint = f2fs_io_type_to_rw_hint(sbi, fio->type, fio->temp); bio->bi_write_stream = f2fs_io_type_to_write_stream(bdev, fio->type, @@ -559,7 +624,7 @@ static void f2fs_set_bio_crypt_ctx(struct bio *bio, const struct inode *inode, * The f2fs garbage collector sets ->encrypted_page when it wants to * read/write raw data without encryption. */ - if (!fio || !fio->encrypted_page) + if (!fio || (!fio->encrypted_page && !fio->is_cache)) fscrypt_set_bio_crypt_ctx(bio, inode, (loff_t)first_idx << inode->i_blkbits, gfp_mask); @@ -769,6 +834,53 @@ void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, __submit_merged_write_cond(sbi, NULL, folio, 0, type, true); } +static bool __has_merged_cache(struct bio *bio, + struct f2fs_cached_block *target) +{ + struct f2fs_cached_block *entry; + + if (!bio) + return false; + + entry = F2FS_BIO(bio)->entry; + + while (entry) { + if (target && entry == target) + return true; + entry = entry->next_entry; + } + return false; +} + +bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, + enum page_type type) +{ + struct f2fs_sb_info *sbi = entry->cache->sbi; + enum temp_type temp; + bool ret = false; + + for (temp = HOT; temp < NR_TEMP_TYPE; temp++) { + enum page_type btype = PAGE_TYPE_OF_BIO(type); + struct f2fs_bio_info *io = sbi->write_io[btype] + temp; + struct f2fs_lock_context lc; + bool merged; + + f2fs_down_read_trace(&io->io_rwsem, &lc); + merged = __has_merged_cache(io->bio, entry); + f2fs_up_read_trace(&io->io_rwsem, &lc); + + if (merged) { + __f2fs_submit_merged_write(sbi, type, temp); + ret = true; + } + + /* TODO: use HOT temp only for meta pages now. */ + if (type >= META) + break; + } + return ret; +} + void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi) { f2fs_submit_merged_write(sbi, DATA); @@ -832,6 +944,8 @@ static bool io_type_is_mergeable(struct f2fs_bio_info *io, if (io->fio.op != fio->op) return false; + if (io->fio.is_cache != fio->is_cache) + return false; return (io->fio.op_flags & mask) == (fio->op_flags & mask); } @@ -1063,6 +1177,34 @@ static bool is_end_zone_blkaddr(struct f2fs_sb_info *sbi, block_t blkaddr) f2fs_blkz_is_seq(sbi, devi, blkaddr) && (blkaddr % sbi->blocks_per_blkz == sbi->blocks_per_blkz - 1); } + +static void f2fs_wait_zone_io_completion(struct f2fs_sb_info *sbi, + struct f2fs_bio_info *io, enum page_type btype) +{ + if (f2fs_sb_has_blkzoned(sbi) && btype < META && io->zone_pending_bio) { + wait_for_completion_io(&io->zone_wait); + bio_put(io->zone_pending_bio); + io->zone_pending_bio = NULL; + io->bi_private = NULL; + } +} + +static void f2fs_submit_zone_io(struct f2fs_sb_info *sbi, + struct f2fs_io_info *fio, struct f2fs_bio_info *io, + enum page_type btype) +{ + if (f2fs_sb_has_blkzoned(sbi) && btype < META && + is_end_zone_blkaddr(sbi, fio->new_blkaddr)) { + bio_get(io->bio); + reinit_completion(&io->zone_wait); + io->bi_private = io->bio->bi_private; + io->bio->bi_private = io; + io->bio->bi_end_io = f2fs_zone_write_end_io; + io->zone_pending_bio = io->bio; + __submit_merged_bio(io); + } + +} #endif void f2fs_submit_page_write(struct f2fs_io_info *fio) @@ -1079,14 +1221,8 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio) f2fs_down_write_trace(&io->io_rwsem, &lc); next: #ifdef CONFIG_BLK_DEV_ZONED - if (f2fs_sb_has_blkzoned(sbi) && btype < META && io->zone_pending_bio) { - wait_for_completion_io(&io->zone_wait); - bio_put(io->zone_pending_bio); - io->zone_pending_bio = NULL; - io->bi_private = NULL; - } + f2fs_wait_zone_io_completion(sbi, io, btype); #endif - if (fio->in_list) { spin_lock(&io->io_lock); if (list_empty(&io->io_list)) { @@ -1141,16 +1277,109 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio) trace_f2fs_submit_folio_write(fio->folio, fio); #ifdef CONFIG_BLK_DEV_ZONED - if (f2fs_sb_has_blkzoned(sbi) && btype < META && - is_end_zone_blkaddr(sbi, fio->new_blkaddr)) { - bio_get(io->bio); - reinit_completion(&io->zone_wait); - io->bi_private = io->bio->bi_private; - io->bio->bi_private = io; - io->bio->bi_end_io = f2fs_zone_write_end_io; - io->zone_pending_bio = io->bio; + f2fs_submit_zone_io(sbi, fio, io, btype); +#endif + + if (fio->in_list) + goto next; +out: + if (is_sbi_flag_set(sbi, SBI_IS_SHUTDOWN) || + !f2fs_is_checkpoint_ready(sbi)) __submit_merged_bio(io); + f2fs_up_write_trace(&io->io_rwsem, &lc); +} + +static void f2fs_bio_add_cache(struct f2fs_io_info *fio, struct bio *bio) +{ + struct f2fs_bio *fbio = F2FS_BIO(bio); + struct f2fs_cached_block *head = fbio->entry; + struct f2fs_cached_block *new = fio->cache_entry; + + new->next_entry = head; + fbio->entry = new; +} + +int f2fs_submit_cache_read(struct f2fs_io_info *fio) +{ + struct f2fs_sb_info *sbi = fio->sbi; + struct f2fs_cached_block *entry = fio->cache_entry; + struct bio *bio; + enum count_type io_type = IS_META_CACHE(entry->cache) ? + F2FS_RD_META : F2FS_RD_NODE; + + if (!f2fs_is_valid_blkaddr(fio->sbi, fio->new_blkaddr, + fio->is_por ? META_POR : (__is_meta_io(fio) ? + META_GENERIC : DATA_GENERIC_ENHANCE))) + return -EFSCORRUPTED; + + bio = __bio_alloc(fio, 1); + + bio_add_virt_nofail(bio, cache_address(entry), sbi->blocksize); + f2fs_bio_add_cache(fio, bio); + inc_page_count(sbi, io_type); + + f2fs_submit_read_bio(sbi, bio, fio->type); + return 0; +} + +void f2fs_submit_cache_write(struct f2fs_io_info *fio) +{ + struct f2fs_sb_info *sbi = fio->sbi; + enum page_type btype = PAGE_TYPE_OF_BIO(fio->type); + struct f2fs_bio_info *io = sbi->write_io[btype] + fio->temp; + struct f2fs_lock_context lc; + struct folio *folio; + + f2fs_bug_on(sbi, is_read_io(fio->op)); + + f2fs_down_write_trace(&io->io_rwsem, &lc); +next: +#ifdef CONFIG_BLK_DEV_ZONED + f2fs_wait_zone_io_completion(sbi, io, btype); +#endif + if (fio->in_list) { + spin_lock(&io->io_lock); + if (list_empty(&io->io_list)) { + spin_unlock(&io->io_lock); + goto out; + } + fio = list_first_entry(&io->io_list, + struct f2fs_io_info, list); + list_del(&fio->list); + spin_unlock(&io->io_lock); } + + verify_fio_blkaddr(fio); + + fio->submitted = 1; + inc_page_count(sbi, F2FS_WB_CP_DATA); + + if (io->bio && + (!io_is_mergeable(sbi, io->bio, io, fio, io->last_block_in_bio, + fio->new_blkaddr))) + __submit_merged_bio(io); +alloc_new: + if (io->bio == NULL) { + io->bio = __bio_alloc(fio, BIO_MAX_VECS); + io->fio = *fio; + } + + folio = cache_folio(fio->cache_entry); + + if (!bio_add_folio(io->bio, folio, sbi->blocksize, + offset_in_folio(folio, cache_address(fio->cache_entry)))) { + f2fs_bug_on(sbi, !F2FS_BIO(io->bio)->entry); + + __submit_merged_bio(io); + goto alloc_new; + } + + f2fs_bio_add_cache(fio, io->bio); + + io->last_block_in_bio = fio->new_blkaddr; + +#ifdef CONFIG_BLK_DEV_ZONED + f2fs_submit_zone_io(sbi, fio, io, btype); #endif if (fio->in_list) goto next; diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 1b96d8718c5c..8413983ea9d5 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -221,6 +221,8 @@ struct f2fs_rwsem { #endif }; +#include "cache.h" + struct f2fs_mount_info { unsigned long long opt; block_t root_reserved_blocks; /* root reserved blocks */ @@ -1370,8 +1372,10 @@ struct f2fs_io_info { unsigned int is_por:1; /* indicate IO is from recovery or not */ unsigned int encrypted:1; /* indicate file is encrypted */ unsigned int meta_gc:1; /* require meta inode GC */ + unsigned int is_cache:1; /* indicate IO is from internal cache */ enum iostat_type io_type; /* io type */ struct writeback_control *io_wbc; /* writeback control */ + struct f2fs_cached_block *cache_entry; struct bio **bio; /* bio for ipu */ sector_t *last_block; /* last block number in bio */ }; @@ -1781,6 +1785,12 @@ struct f2fs_gc_kthread { unsigned int boost_gc_greedy; }; +struct f2fs_bio { + struct work_struct work; + struct f2fs_cached_block *entry; + struct bio bio; +}; + struct f2fs_sb_info { struct super_block *sb; /* pointer to VFS super block */ struct proc_dir_entry *s_proc; /* proc entry */ @@ -2323,6 +2333,16 @@ static inline bool is_node_folio(struct folio *folio) return folio->mapping == NODE_MAPPING(F2FS_F_SB(folio)); } +static inline struct f2fs_bio *F2FS_BIO(struct bio *bio) +{ + return container_of(bio, struct f2fs_bio, bio); +} + +static inline bool f2fs_is_cache_bio(struct bio *bio) +{ + return F2FS_BIO(bio)->entry != NULL; +} + static inline bool is_sbi_flag_set(struct f2fs_sb_info *sbi, unsigned int type) { return test_bit(type, &sbi->s_flag); @@ -4237,6 +4257,8 @@ void f2fs_submit_merged_write_cond(struct f2fs_sb_info *sbi, nid_t ino, enum page_type type); void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, struct folio *folio, enum page_type type); +bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, + enum page_type type); void f2fs_submit_merged_ipu_write(struct f2fs_sb_info *sbi, struct bio **bio, struct folio *folio); void f2fs_submit_all_merged_ipu_writes(struct f2fs_sb_info *sbi); @@ -4244,6 +4266,8 @@ void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi); int f2fs_submit_page_bio(struct f2fs_io_info *fio); int f2fs_merge_page_bio(struct f2fs_io_info *fio); void f2fs_submit_page_write(struct f2fs_io_info *fio); +int f2fs_submit_cache_read(struct f2fs_io_info *fio); +void f2fs_submit_cache_write(struct f2fs_io_info *fio); struct block_device *f2fs_target_device(struct f2fs_sb_info *sbi, block_t blk_addr, sector_t *sector); int f2fs_target_device_index(struct f2fs_sb_info *sbi, block_t blkaddr); diff --git a/fs/f2fs/iostat.h b/fs/f2fs/iostat.h index 2025225b5bed..61c6bc8e3119 100644 --- a/fs/f2fs/iostat.h +++ b/fs/f2fs/iostat.h @@ -60,6 +60,13 @@ static inline struct bio_post_read_ctx *get_post_read_ctx(struct bio *bio) return iostat_ctx->post_read_ctx; } +static inline void iostat_set_post_read_ctx(struct bio *bio, void *ctx) +{ + struct bio_iostat_ctx *iostat_ctx = bio->bi_private; + + iostat_ctx->post_read_ctx = ctx; +} + extern void iostat_update_and_unbind_ctx(struct bio *bio); extern void iostat_alloc_and_bind_ctx(struct f2fs_sb_info *sbi, struct bio *bio, struct bio_post_read_ctx *ctx); @@ -81,6 +88,10 @@ static inline struct bio_post_read_ctx *get_post_read_ctx(struct bio *bio) { return bio->bi_private; } +static inline void iostat_set_post_read_ctx(struct bio *bio, void *ctx) +{ + bio->bi_private = ctx; +} static inline int f2fs_init_iostat_processing(void) { return 0; } static inline void f2fs_destroy_iostat_processing(void) {} static inline int f2fs_init_iostat(struct f2fs_sb_info *sbi) { return 0; } -- 2.49.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: [f2fs-dev] [PATCH v1 01/12] f2fs: cache: implement metadata cache 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel @ 2026-08-20 5:08 ` Jaegeuk Kim via Linux-f2fs-devel -1 siblings, 0 replies; 34+ messages in thread From: Jaegeuk Kim @ 2026-08-20 5:08 UTC (permalink / raw) To: Chao Yu; +Cc: linux-kernel, linux-f2fs-devel On 08/20, Chao Yu via Linux-f2fs-devel wrote: > This patch introduces the core metadata block caching infrastructure to > manage f2fs metadata independently of the page cache. > > It implements: > - core cache APIs: get, create, put, drop, backed by a radix tree and > a single global LRU list. > - support multiple status of cached block: LOCKED, UPTODATE, DIRTY, > WRITEBACK, INLINE. > - internal bio based read/write helpers with adjacent block vector merging. > > Signed-off-by: Chao Yu <chao@kernel.org> > --- > fs/f2fs/Makefile | 2 +- > fs/f2fs/cache.c | 531 +++++++++++++++++++++++++++++++++++++++++++++++ > fs/f2fs/cache.h | 179 ++++++++++++++++ > fs/f2fs/data.c | 283 ++++++++++++++++++++++--- > fs/f2fs/f2fs.h | 24 +++ > fs/f2fs/iostat.h | 11 + > 6 files changed, 1002 insertions(+), 28 deletions(-) > create mode 100644 fs/f2fs/cache.c > create mode 100644 fs/f2fs/cache.h > > diff --git a/fs/f2fs/Makefile b/fs/f2fs/Makefile > index 8a7322d229e4..fbf49c30b066 100644 > --- a/fs/f2fs/Makefile > +++ b/fs/f2fs/Makefile > @@ -3,7 +3,7 @@ obj-$(CONFIG_F2FS_FS) += f2fs.o > > f2fs-y := dir.o file.o inode.o namei.o hash.o super.o inline.o > f2fs-y += checkpoint.o gc.o data.o node.o segment.o recovery.o > -f2fs-y += shrinker.o extent_cache.o sysfs.o > +f2fs-y += shrinker.o extent_cache.o sysfs.o cache.o > f2fs-$(CONFIG_F2FS_STAT_FS) += debug.o > f2fs-$(CONFIG_F2FS_FS_XATTR) += xattr.o > f2fs-$(CONFIG_F2FS_FS_POSIX_ACL) += acl.o > diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c > new file mode 100644 > index 000000000000..08bc658166f7 > --- /dev/null > +++ b/fs/f2fs/cache.c > @@ -0,0 +1,531 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * Copyright (c) 2026 Google LLC > + * Author: Chao Yu <chaseyu@google.com> > + */ > +#include "linux/spinlock.h" #include <linux/spinlock.h>? May need to Move below? > +#include <linux/fs.h> > +#include <linux/f2fs_fs.h> > +#include <linux/radix-tree.h> > +#include <linux/slab.h> > +#include <linux/list.h> > +#include <linux/pagemap.h> > +#include <linux/kthread.h> > +#include <linux/freezer.h> > +#include <linux/delay.h> > +#include "f2fs.h" > +#include "cache.h" > +#include "node.h" > +#include "segment.h" > + > +void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, > + enum page_type type) > +{ > + /* in case the entry was truncated or on-going shrink */ > + if (!entry->cache) > + return; > + > + if (!f2fs_cache_test_writeback(entry)) > + return; > + > + /* submit cached bio */ > + f2fs_submit_merged_write_cache(entry, type); > + > + wait_on_bit_io(&entry->state, F2FS_BLOCK_WRITEBACK, > + TASK_UNINTERRUPTIBLE); > +} > + > +void f2fs_cache_wait_writeback(struct f2fs_cached_block *entry) > +{ > + /* in case the entry was truncated or on-going shrink */ > + if (!entry->cache) > + return; > + > + f2fs_cache_wait_writeback_cond(entry, > + IS_META_CACHE(entry->cache) ? META : NODE); > +} > + > +static void f2fs_cache_update_tag(struct f2fs_cached_block *entry, unsigned int src, > + unsigned int dst) f2fs_cache_update_tag(entry, clear_from, set_to); > +{ > + struct f2fs_cached_block_list *cache = entry->cache; > + unsigned long flags; > + > + spin_lock_irqsave(&cache->tree_lock, flags); > + if (src) if (clear_fom != F2FS_CACHE_TAG_NONE) > + radix_tree_tag_clear(&cache->root, entry->index, src); > + if (dst) if (set_to != F2FS_CACHE_TAG_NONE) > + radix_tree_tag_set(&cache->root, entry->index, dst); > + spin_unlock_irqrestore(&cache->tree_lock, flags); > +} > + > +bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry) > +{ > + struct f2fs_cached_block_list *cache = entry->cache; > + > + if (!f2fs_cache_test_uptodate(entry)) > + f2fs_cache_set_uptodate(entry); f2fs_cache_set_uptodate(entry); > + > +#ifdef CONFIG_F2FS_CHECK_FS > + if (cache->type == F2FS_NODE_CACHE && IS_INODE(cache_folio(entry))) > + f2fs_inode_chksum_set(cache->sbi, cache_folio(entry)); > +#endif > + > + if (f2fs_cache_test_dirty(entry)) > + return false; > + > + if (!f2fs_cache_test_and_set_dirty(entry)) { > + enum count_type type = IS_META_CACHE(cache) ? > + F2FS_DIRTY_META : F2FS_DIRTY_NODES; > + > + f2fs_cache_update_tag(entry, 0, F2FS_CACHE_TAG_DIRTY); f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_NONE, F2FS_CACHE_TAG_DIRTY); > + inc_page_count(cache->sbi, type); > + return true; > + } > + > + return false; > +} > + > +bool f2fs_clear_cache_dirty(struct f2fs_cached_block *entry) > +{ > + if (!f2fs_cache_test_dirty(entry)) > + return false; > + > + f2fs_cache_clear_dirty(entry); > + return true; > +} > + > +static void __f2fs_drop_cache_dirty(struct f2fs_cached_block *entry, bool force) static void __drop_cache_dirty(struct f2fs_cached_block *entry, bool force) > +{ > + > + struct f2fs_cached_block_list *cache = entry->cache; > + enum count_type type = IS_META_CACHE(cache) ? > + F2FS_DIRTY_META : F2FS_DIRTY_NODES; > + > + f2fs_cache_clear_uptodate(entry); > + > + if (!force && !f2fs_clear_cache_dirty(entry)) > + return; > + > + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_DIRTY, 0); > + dec_page_count(cache->sbi, type); > +} > + > +void f2fs_drop_cache_dirty(struct f2fs_cached_block *entry) > +{ > + __f2fs_drop_cache_dirty(entry, false); > +} > + > +void f2fs_force_clear_cache_dirty(struct f2fs_cached_block *entry) > +{ > + __f2fs_drop_cache_dirty(entry, true); > +} > + > +void f2fs_start_cache_writeback(struct f2fs_cached_block *entry) > +{ > + f2fs_cache_set_writeback(entry); > + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_DIRTY, > + F2FS_CACHE_TAG_WRITEBACK); > +} > + > +void f2fs_end_cache_writeback(struct f2fs_cached_block *entry) > +{ > + /* > + * should call f2fs_cache_update_tag() before clearing writeback bit, > + * in case f2fs_truncate_cache() set entry->cache to NULL. > + */ > + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_WRITEBACK, 0); > + clear_and_wake_up_bit(F2FS_BLOCK_WRITEBACK, &entry->state); > +} > + > +static int f2fs_cache_refcount(struct f2fs_cached_block *entry) > +{ > + return atomic_read(&entry->refcount); > +} > + > +static void __f2fs_free_cache(struct f2fs_cached_block *entry) > +{ > + kfree(entry->data); > + kfree(entry); > +} > + > +static void f2fs_free_cache(struct f2fs_cached_block *entry) > +{ > + WARN_ON_ONCE(!list_empty(&entry->list)); > + WARN_ON_ONCE(f2fs_cache_refcount(entry)); > + __f2fs_free_cache(entry); > +} > + > +void f2fs_cache_get(struct f2fs_cached_block *entry) > +{ > + atomic_inc(&entry->refcount); > +} > + > +static bool f2fs_cache_put(struct f2fs_cached_block *entry) > +{ > + WARN_ON_ONCE(!f2fs_cache_refcount(entry)); > + if (atomic_dec_and_test(&entry->refcount)) { > + f2fs_free_cache(entry); > + return true; > + } > + return false; > +} > + > +static struct f2fs_cached_block *f2fs_create_cache( > + struct f2fs_cached_block_list *cache, > + unsigned long index, bool nofail) > +{ > + struct f2fs_cached_block *entry; > + unsigned int flags = GFP_NOFS; > + > + if (nofail) > + flags |= __GFP_NOFAIL; > + > + entry = kzalloc_obj(*entry, flags); > + if (!entry) > + return ERR_PTR(-ENOMEM); > + > + entry->data = kzalloc(cache->sbi->blocksize, flags); We don't need kzalloc() since we have an uptodate flag. > + if (!entry->data) { > + kfree(entry); > + return ERR_PTR(-ENOMEM); > + } > + > + entry->index = index; > + > + atomic_set(&entry->refcount, 0); > + entry->next_entry = NULL; > + INIT_LIST_HEAD(&entry->list); > + > + entry->cache = cache; > + > + return entry; > +} > + > +static struct f2fs_cached_block *f2fs_insert_cache( > + struct f2fs_cached_block_list *cache, > + unsigned long index, > + struct f2fs_cached_block *new) > +{ > + struct f2fs_cached_block *e; > + int ret; > + unsigned long flags; > + > + ret = radix_tree_preload(GFP_NOFS | __GFP_NOFAIL); > + f2fs_bug_on(cache->sbi, ret); > + > + spin_lock(&cache->list_lock); > + spin_lock_irqsave(&cache->tree_lock, flags); > + e = radix_tree_lookup(&cache->root, index); > + if (!e) { > + e = new; > + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(e)); > + > + ret = radix_tree_insert(&cache->root, index, e); > + f2fs_bug_on(cache->sbi, ret); > + > + /* radix tree referenced cache entry */ > + f2fs_cache_get(e); > + f2fs_bug_on(cache->sbi, !list_empty(&e->list)); > + list_add_tail(&e->list, &cache->lru_list); > + cache->num_entries++; > + } > + f2fs_cache_get(e); > + spin_unlock_irqrestore(&cache->tree_lock, flags); > + spin_unlock(&cache->list_lock); > + radix_tree_preload_end(); > + > + if (new != e) { > + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(new)); > + __f2fs_free_cache(new); > + } > + > + return e; > +} > + > +struct f2fs_cached_block *f2fs_find_cache( > + struct f2fs_cached_block_list *cache, > + unsigned long index) > +{ > + struct f2fs_cached_block *entry; > + unsigned long flags; > + > + spin_lock(&cache->list_lock); > + spin_lock_irqsave(&cache->tree_lock, flags); > + entry = radix_tree_lookup(&cache->root, index); > + if (entry) { > + f2fs_bug_on(cache->sbi, !f2fs_cache_refcount(entry)); > + f2fs_cache_get(entry); > + if (!list_empty(&entry->list)) > + list_move_tail(&entry->list, &cache->lru_list); > + > + } else { > + entry = ERR_PTR(-ENOENT); > + } > + spin_unlock_irqrestore(&cache->tree_lock, flags); > + spin_unlock(&cache->list_lock); > + > + return entry; > +} > + > +struct f2fs_cached_block *f2fs_grab_cache( > + struct f2fs_cached_block_list *cache, > + unsigned long index, int flags) > + > +{ > + struct f2fs_cached_block *entry, *new; > + bool create = flags & F2FS_CACHE_CREATE; > + bool nofail = flags & F2FS_CACHE_NOFAIL; > + bool lock = flags & F2FS_CACHE_LOCK; > + > +repeat: > + entry = f2fs_find_cache(cache, index); > + if (!IS_ERR(entry)) > + goto found; > + > + if (!create) > + return ERR_PTR(-ENOENT); > + > + new = f2fs_create_cache(cache, index, nofail); > + if (IS_ERR(new)) > + return new; > + > + entry = f2fs_insert_cache(cache, index, new); > +found: > + if (lock) { > + f2fs_lock_cache(entry); > + /* has been truncated */ > + if (entry->cache != cache) { > + f2fs_put_cache(entry, true); > + goto repeat; > + } > + } > + return entry; > +} > + > +bool f2fs_trylock_cache(struct f2fs_cached_block *entry) > +{ > + return !test_and_set_bit(F2FS_BLOCK_LOCKED, &entry->state); > +} > + > +void f2fs_lock_cache(struct f2fs_cached_block *entry) > +{ > + wait_on_bit_lock(&entry->state, F2FS_BLOCK_LOCKED, > + TASK_UNINTERRUPTIBLE); > +} > + > +void f2fs_unlock_cache(struct f2fs_cached_block *entry) > +{ > + clear_and_wake_up_bit(F2FS_BLOCK_LOCKED, &entry->state); > +} > + > +bool f2fs_put_cache(struct f2fs_cached_block *entry, bool unlock) > +{ > + if (IS_ERR_OR_NULL(entry)) > + return false; > + if (unlock) > + f2fs_unlock_cache(entry); > + return f2fs_cache_put(entry); > +} > + > +unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache, > + struct f2fs_cached_block **entries, > + pgoff_t *first_index, unsigned int max_nr, > + int tag) > +{ > + unsigned long flags; > + int nr, i; > + > + spin_lock_irqsave(&cache->tree_lock, flags); > + nr = radix_tree_gang_lookup_tag(&cache->root, (void **)entries, > + *first_index, max_nr, tag); > + if (!nr) > + goto out; > + > + for (i = 0; i < nr; i++) > + f2fs_cache_get(entries[i]); > + *first_index = entries[nr - 1]->index + 1; > +out: > + spin_unlock_irqrestore(&cache->tree_lock, flags); > + return nr; > +} > + > +void f2fs_cache_gang_release(struct f2fs_cached_block **entries, > + unsigned int nr_entries) > +{ > + int i; > + > + for (i = 0; i < nr_entries; i++) > + f2fs_put_cache(entries[i], false); > +} > + > +void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache) > +{ > + unsigned long index = 0; > + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; > + int nr, i; > + > +next: > + nr = f2fs_cache_gang_lookup_tag(cache, entries, &index, > + F2FS_ONSTACK_CACHES, F2FS_CACHE_TAG_WRITEBACK); > + if (!nr) > + return; > + > + for (i = 0; i < nr; i++) > + f2fs_cache_wait_writeback(entries[i]); > + f2fs_cache_gang_release(entries, nr); > + goto next; > +} > + > +static void f2fs_do_truncate_cache(struct f2fs_cached_block *entry, > + bool drop_dirty) > +{ > + struct f2fs_cached_block_list *cache = entry->cache; > + unsigned long flags; > + > + if (!drop_dirty && > + (f2fs_cache_test_dirty(entry) || > + f2fs_cache_test_writeback(entry))) > + return; > + > + f2fs_cache_wait_writeback(entry); > + f2fs_drop_cache_dirty(entry); > + > + spin_lock(&cache->list_lock); > + spin_lock_irqsave(&cache->tree_lock, flags); > + > + f2fs_bug_on(cache->sbi, !entry->cache); > + if (!radix_tree_delete(&cache->root, entry->index)) > + f2fs_bug_on(cache->sbi, !entry->cache); > + > + entry->cache = NULL; > + cache->num_entries--; > + > + atomic_dec(&entry->refcount); > + f2fs_bug_on(cache->sbi, !f2fs_cache_refcount(entry)); > + > + f2fs_bug_on(cache->sbi, list_empty(&entry->list)); > + list_del_init(&entry->list); > + > + spin_unlock_irqrestore(&cache->tree_lock, flags); > + spin_unlock(&cache->list_lock); > +} > + > +static void f2fs_truncate_cache(struct f2fs_cached_block *entry, > + bool drop_dirty) > +{ > + f2fs_lock_cache(entry); > + if (entry->cache) > + f2fs_do_truncate_cache(entry, drop_dirty); > + f2fs_unlock_cache(entry); > +} > + > +static void f2fs_drop_cache(struct f2fs_cached_block_list *cache, > + block_t blkaddr, bool drop_dirty) > +{ > + struct f2fs_cached_block *entry; > + > + entry = f2fs_find_cache(cache, blkaddr); > + if (IS_ERR(entry)) > + return; > + > + f2fs_truncate_cache(entry, drop_dirty); > + f2fs_put_cache(entry, false); > +} > + > +void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache, > + unsigned long start, unsigned long len, bool drop_dirty) > +{ > + unsigned long index = start; > + unsigned long end = (ULONG_MAX - start < len) ? > + ULONG_MAX : (start + len); > + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; > + unsigned long flags; > + int nr, i; > + > + if (len == 1) > + return f2fs_drop_cache(cache, index, drop_dirty); > + > +next: > + spin_lock_irqsave(&cache->tree_lock, flags); > + nr = radix_tree_gang_lookup(&cache->root, (void **)entries, index, > + min((unsigned long)F2FS_ONSTACK_CACHES, end - index)); > + if (!nr) > + goto out_unlock; > + > + for (i = 0; i < nr; i++) { > + struct f2fs_cached_block *entry = entries[i]; > + > + if (entry->index >= end) { > + nr = i; > + break; > + } > + f2fs_cache_get(entry); > + } > +out_unlock: > + spin_unlock_irqrestore(&cache->tree_lock, flags); > + if (!nr) > + return; > + > + for (i = 0; i < nr; i++) { > + struct f2fs_cached_block *entry = entries[i]; > + > + index = entry->index + 1; > + > + f2fs_truncate_cache(entry, drop_dirty); > + } > + f2fs_cache_gang_release(entries, nr); > + > + if (index < end) > + goto next; > +} > + > +int f2fs_init_cache(struct f2fs_sb_info *sbi, > + struct f2fs_cached_block_list *cache, > + enum f2fs_cache_type type) > +{ > + cache->sbi = sbi; > + cache->type = type; > + INIT_RADIX_TREE(&cache->root, GFP_ATOMIC); > + spin_lock_init(&cache->tree_lock); > + spin_lock_init(&cache->list_lock); > + INIT_LIST_HEAD(&cache->lru_list); > + cache->num_entries = 0; > + > + return 0; > +} > + > +void f2fs_destroy_cache(struct f2fs_cached_block_list *cache) > +{ > + struct list_head *head = &cache->lru_list; > + struct f2fs_cached_block *entry; > + unsigned long flags; > + > + f2fs_cache_wait_on_all_writeback(cache); > +next: > + spin_lock(&cache->list_lock); > + if (list_empty(head)) { > + spin_unlock(&cache->list_lock); > + return; > + } > + entry = list_first_entry(head, struct f2fs_cached_block, list); > + > + spin_lock_irqsave(&cache->tree_lock, flags); > + radix_tree_delete(&cache->root, entry->index); > + cache->num_entries--; > + list_del_init(&entry->list); > + spin_unlock_irqrestore(&cache->tree_lock, flags); > + > + spin_unlock(&cache->list_lock); > + > + /* wait on read cache IO */ > + f2fs_lock_cache(entry); > + /* wait on write cache IO */ > + f2fs_cache_wait_writeback(entry); > + f2fs_bug_on(cache->sbi, f2fs_cache_test_dirty(entry)); > + f2fs_bug_on(cache->sbi, f2fs_cache_test_writeback(entry)); > + f2fs_bug_on(cache->sbi, !list_empty(&entry->list)); > + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(entry) != 1); > + f2fs_put_cache(entry, true); > + goto next; > +} > diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h > new file mode 100644 > index 000000000000..686a974008ae > --- /dev/null > +++ b/fs/f2fs/cache.h > @@ -0,0 +1,179 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * Copyright (c) 2026 Google LLC > + * Author: Chao Yu <chaseyu@google.com> > + */ > +#ifndef _LINUX_F2FS_CACHE_H > +#define _LINUX_F2FS_CACHE_H > + > +#include <linux/pagemap.h> > +#include <linux/mm.h> > +#include <linux/list.h> > +#include <linux/radix-tree.h> > +#include <linux/spinlock.h> > +#include <linux/wait.h> > +#include <linux/types.h> > + > +struct f2fs_rwsem; > +struct f2fs_io_info; > +enum page_type; > + > +/* Represents a single cached block (meta, node or compress) */ > +struct f2fs_cached_block { > + struct list_head list; /* LRU list head */ > + struct f2fs_cached_block_list *cache; /* parent cache list */ > + union { > + /* chain for merged BIO */ > + struct f2fs_cached_block *next_entry; > + nid_t ino; /* inode number for compress cache */ > + }; > + unsigned long index; /* key in radix tree, (meta/compress: pba, node: nid) */ > + unsigned long state; /* cache entry state (e.g., Dirty, UpToDate) */ > + void *data; /* blocksize-aligned memory (4KB or 16KB) */ > + atomic_t refcount; /* reference count */ > +}; > + > +struct f2fs_sb_info; > + > +enum f2fs_cache_type { > + F2FS_META_CACHE, > + F2FS_NODE_CACHE, > +}; > + > +/* Main cache control structure (per sb_info) */ > +struct f2fs_cached_block_list { > + struct f2fs_sb_info *sbi; /* Pointer to f2fs_sb_info */ > + struct radix_tree_root root; /* Radix tree for cache lookup */ > + spinlock_t tree_lock; /* Lock for radix tree */ > + struct list_head lru_list; /* Single global LRU list */ > + spinlock_t list_lock; /* Lock for LRU list */ > + enum f2fs_cache_type type; /* Cache type (Node or Meta) */ > + unsigned long num_entries; /* Current number of entries */ > +}; > + > +#define IS_META_CACHE(cache) (cache->type == F2FS_META_CACHE) > + > +/* Flags for f2fs_cached_block state */ > +enum f2fs_cached_state { > + F2FS_BLOCK_LOCKED, /* cache entry is locked */ > + F2FS_BLOCK_UPTODATE, /* cache data is valid */ > + F2FS_BLOCK_DIRTY, /* cache data is dirty, need to writeback the data */ > + F2FS_BLOCK_WRITEBACK, /* cache data is writeback state */ > + F2FS_BLOCK_INLINE_DATA, /* indicate inline data */ > +}; > + > +enum { > + __F2FS_CACHE_CREATE, /* create the cache if there is no cache entry */ > + __F2FS_CACHE_LOCK, /* get and lock the cache entry */ > + __F2FS_CACHE_NOFAIL, /* do not allow failure */ > +}; > + > +enum f2fs_cache_request_flag { > + F2FS_CACHE_CREATE = 1 << __F2FS_CACHE_CREATE, > + F2FS_CACHE_LOCK = 1 << __F2FS_CACHE_LOCK, > + F2FS_CACHE_NOFAIL = 1 << __F2FS_CACHE_NOFAIL, > +}; > + > +#define F2FS_CACHE_LOCK_CREATE (F2FS_CACHE_LOCK | F2FS_CACHE_CREATE) > + > +#define F2FS_ONSTACK_CACHES (32) > + > +#define F2FS_CACHE_FLAG_TEST_FUNC(name, flagname) \ > +static inline bool f2fs_cache_test_##name( \ > + const struct f2fs_cached_block *entry) \ > +{ \ > + return test_bit(F2FS_BLOCK_##flagname, &entry->state); \ > +} \ > + > +#define F2FS_CACHE_FLAG_SET_FUNC(name, flagname) \ > +static inline void f2fs_cache_set_##name( \ > + struct f2fs_cached_block *entry) \ > +{ \ > + set_bit(F2FS_BLOCK_##flagname, &entry->state); \ > +} \ > + > +#define F2FS_CACHE_FLAG_CLEAR_FUNC(name, flagname) \ > +static inline void f2fs_cache_clear_##name( \ > + struct f2fs_cached_block *entry) \ > +{ \ > + clear_bit(F2FS_BLOCK_##flagname, &entry->state); \ > +} \ > + > +#define F2FS_CACHE_FLAG_TEST_AND_SET_FUNC(name, flagname) \ > +static inline bool f2fs_cache_test_and_set_##name( \ > + struct f2fs_cached_block *entry) \ > +{ \ > + return test_and_set_bit(F2FS_BLOCK_##flagname, &entry->state); \ > +} \ > + > +F2FS_CACHE_FLAG_TEST_FUNC(locked, LOCKED); > +F2FS_CACHE_FLAG_SET_FUNC(locked, LOCKED); > +F2FS_CACHE_FLAG_CLEAR_FUNC(locked, LOCKED); > + > +F2FS_CACHE_FLAG_TEST_FUNC(uptodate, UPTODATE); > +F2FS_CACHE_FLAG_SET_FUNC(uptodate, UPTODATE); > +F2FS_CACHE_FLAG_CLEAR_FUNC(uptodate, UPTODATE); > + > +F2FS_CACHE_FLAG_TEST_FUNC(dirty, DIRTY); > +F2FS_CACHE_FLAG_SET_FUNC(dirty, DIRTY); > +F2FS_CACHE_FLAG_CLEAR_FUNC(dirty, DIRTY); > +F2FS_CACHE_FLAG_TEST_AND_SET_FUNC(dirty, DIRTY); > + > +F2FS_CACHE_FLAG_TEST_FUNC(writeback, WRITEBACK); > +F2FS_CACHE_FLAG_SET_FUNC(writeback, WRITEBACK); > +F2FS_CACHE_FLAG_CLEAR_FUNC(writeback, WRITEBACK); > + > +F2FS_CACHE_FLAG_TEST_FUNC(inline, INLINE_DATA); > +F2FS_CACHE_FLAG_SET_FUNC(inline, INLINE_DATA); > +F2FS_CACHE_FLAG_CLEAR_FUNC(inline, INLINE_DATA); > + > +static inline void *cache_address(const struct f2fs_cached_block *entry) > +{ > + return entry->data; > +} > + > +#define CACHED_NODE(entry) ((struct f2fs_node *)(cache_address(entry))) > + > +static inline struct folio *cache_folio(const struct f2fs_cached_block *entry) > +{ > + return virt_to_folio(entry->data); > +} > + > +int f2fs_init_cache(struct f2fs_sb_info *sbi, > + struct f2fs_cached_block_list *cache, > + enum f2fs_cache_type type); > +void f2fs_destroy_cache(struct f2fs_cached_block_list *cache); > +void f2fs_cache_get(struct f2fs_cached_block *entry); > +struct f2fs_cached_block *f2fs_find_cache( > + struct f2fs_cached_block_list *cache, > + unsigned long index); > +#define F2FS_CACHE_TAG_NONE 0 > +#define F2FS_CACHE_TAG_DIRTY 1 > +#define F2FS_CACHE_TAG_WRITEBACK 2 > + > +bool f2fs_trylock_cache(struct f2fs_cached_block *entry); > +void f2fs_lock_cache(struct f2fs_cached_block *entry); > +void f2fs_unlock_cache(struct f2fs_cached_block *entry); > +bool f2fs_put_cache(struct f2fs_cached_block *entry, bool unlock); > +bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry); > +bool f2fs_clear_cache_dirty(struct f2fs_cached_block *entry); > +void f2fs_drop_cache_dirty(struct f2fs_cached_block *entry); > +void f2fs_force_clear_cache_dirty(struct f2fs_cached_block *entry); > +void f2fs_start_cache_writeback(struct f2fs_cached_block *entry); > +void f2fs_end_cache_writeback(struct f2fs_cached_block *entry); > +unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache, > + struct f2fs_cached_block **results, pgoff_t *first_index, > + unsigned int max_items, int tag); > +void f2fs_cache_gang_release(struct f2fs_cached_block **entries, > + unsigned int nr_entries); > +int f2fs_writeback_cache(struct f2fs_cached_block_list *cache, bool sync); > +void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache); > +void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, > + enum page_type type); > +void f2fs_cache_wait_writeback(struct f2fs_cached_block *entry); > +struct f2fs_cached_block *f2fs_grab_cache(struct f2fs_cached_block_list *cache, > + unsigned long index, int flags); > +void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache, > + unsigned long start, unsigned long len, bool drop_dirty); > + > +#endif /* _LINUX_F2FS_CACHE_H */ > diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c > index 6ae0eb37d20f..09474790035b 100644 > --- a/fs/f2fs/data.c > +++ b/fs/f2fs/data.c > @@ -41,11 +41,6 @@ struct f2fs_folio_state { > unsigned int read_pages_pending; > }; > > -struct f2fs_bio { > - struct work_struct work; > - struct bio bio; > -}; > - > #define F2FS_BIO_POOL_SIZE NR_CURSEG_TYPE > > int __init f2fs_init_bioset(void) > @@ -69,7 +64,6 @@ bool f2fs_is_cp_guaranteed(const struct folio *folio) > return folio_test_f2fs_gcing(fscrypt_pagecache_folio(folio)); > > inode = mapping->host; > - sbi = F2FS_I_SB(inode); > > if (inode->i_ino == F2FS_META_INO(sbi) || > inode->i_ino == F2FS_NODE_INO(sbi) || > @@ -437,6 +431,65 @@ static void f2fs_write_end_io(struct bio *bio) > } > } > > +static void f2fs_cache_read_end_io(struct bio *bio) > +{ > + struct f2fs_cached_block *entry = F2FS_BIO(bio)->entry; > + struct f2fs_sb_info *sbi = entry->cache->sbi; > + enum count_type io_type = IS_META_CACHE(entry->cache) ? > + F2FS_RD_META : F2FS_RD_NODE; > + struct f2fs_cached_block *next; > + > + iostat_update_and_unbind_ctx(bio); > + > + if (time_to_inject(sbi, FAULT_READ_IO)) > + bio->bi_status = BLK_STS_IOERR; > + > + while (entry) { > + next = entry->next_entry; > + entry->next_entry = NULL; > + > + if (bio->bi_status == BLK_STS_OK) > + f2fs_cache_set_uptodate(entry); > + > + dec_page_count(sbi, io_type); > + > + f2fs_unlock_cache(entry); > + entry = next; > + } > + bio_put(bio); > +} > + > +static void f2fs_cache_write_end_io(struct bio *bio) > +{ > + struct f2fs_cached_block *entry = F2FS_BIO(bio)->entry; > + struct f2fs_sb_info *sbi = entry->cache->sbi; > + struct f2fs_cached_block *next; > + > + iostat_update_and_unbind_ctx(bio); > + > + if (time_to_inject(sbi, FAULT_WRITE_IO)) > + bio->bi_status = BLK_STS_IOERR; > + > + if (bio->bi_status != BLK_STS_OK) > + f2fs_stop_checkpoint(sbi, true, > + STOP_CP_REASON_WRITE_FAIL); > + > + while (entry) { > + next = entry->next_entry; > + entry->next_entry = NULL; > + > + dec_page_count(sbi, F2FS_WB_CP_DATA); > + > + if (!get_pages(sbi, F2FS_WB_CP_DATA) && > + wq_has_sleeper(&sbi->cp_wait)) > + wake_up(&sbi->cp_wait); > + > + f2fs_end_cache_writeback(entry); > + entry = next; > + } > + bio_put(bio); > +} > + > #ifdef CONFIG_BLK_DEV_ZONED > static void f2fs_zone_write_end_io(struct bio *bio) > { > @@ -444,7 +497,10 @@ static void f2fs_zone_write_end_io(struct bio *bio) > > bio->bi_private = io->bi_private; > complete(&io->zone_wait); > - f2fs_write_end_io(bio); > + if (io->fio.is_cache) > + f2fs_cache_write_end_io(bio); > + else > + f2fs_write_end_io(bio); > } > #endif > > @@ -531,12 +587,21 @@ static struct bio *__bio_alloc(struct f2fs_io_info *fio, int npages) > fio->op | fio->op_flags | f2fs_io_flags(fio), > GFP_NOIO, &f2fs_bioset); > bio->bi_iter.bi_sector = sector; > + F2FS_BIO(bio)->entry = NULL; > + bio->bi_private = NULL; > if (is_read_io(fio->op)) { > - bio->bi_end_io = f2fs_read_end_io; > - bio->bi_private = NULL; > + if (fio->is_cache) > + bio->bi_end_io = f2fs_cache_read_end_io; > + else > + bio->bi_end_io = f2fs_read_end_io; > } else { > - bio->bi_end_io = f2fs_write_end_io; > - bio->bi_private = sbi; > + if (fio->is_cache) { > + bio->bi_end_io = f2fs_cache_write_end_io; > + } else { > + bio->bi_end_io = f2fs_write_end_io; > + bio->bi_private = sbi; > + } > + > bio->bi_write_hint = f2fs_io_type_to_rw_hint(sbi, > fio->type, fio->temp); > bio->bi_write_stream = f2fs_io_type_to_write_stream(bdev, fio->type, > @@ -559,7 +624,7 @@ static void f2fs_set_bio_crypt_ctx(struct bio *bio, const struct inode *inode, > * The f2fs garbage collector sets ->encrypted_page when it wants to > * read/write raw data without encryption. > */ > - if (!fio || !fio->encrypted_page) > + if (!fio || (!fio->encrypted_page && !fio->is_cache)) > fscrypt_set_bio_crypt_ctx(bio, inode, > (loff_t)first_idx << inode->i_blkbits, > gfp_mask); > @@ -769,6 +834,53 @@ void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, > __submit_merged_write_cond(sbi, NULL, folio, 0, type, true); > } > > +static bool __has_merged_cache(struct bio *bio, > + struct f2fs_cached_block *target) > +{ > + struct f2fs_cached_block *entry; > + > + if (!bio) > + return false; > + > + entry = F2FS_BIO(bio)->entry; > + > + while (entry) { > + if (target && entry == target) > + return true; > + entry = entry->next_entry; > + } > + return false; > +} > + > +bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, > + enum page_type type) > +{ > + struct f2fs_sb_info *sbi = entry->cache->sbi; > + enum temp_type temp; > + bool ret = false; > + > + for (temp = HOT; temp < NR_TEMP_TYPE; temp++) { > + enum page_type btype = PAGE_TYPE_OF_BIO(type); > + struct f2fs_bio_info *io = sbi->write_io[btype] + temp; > + struct f2fs_lock_context lc; > + bool merged; > + > + f2fs_down_read_trace(&io->io_rwsem, &lc); > + merged = __has_merged_cache(io->bio, entry); > + f2fs_up_read_trace(&io->io_rwsem, &lc); > + > + if (merged) { > + __f2fs_submit_merged_write(sbi, type, temp); > + ret = true; > + } > + > + /* TODO: use HOT temp only for meta pages now. */ > + if (type >= META) > + break; > + } > + return ret; > +} > + > void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi) > { > f2fs_submit_merged_write(sbi, DATA); > @@ -832,6 +944,8 @@ static bool io_type_is_mergeable(struct f2fs_bio_info *io, > > if (io->fio.op != fio->op) > return false; > + if (io->fio.is_cache != fio->is_cache) > + return false; > return (io->fio.op_flags & mask) == (fio->op_flags & mask); > } > > @@ -1063,6 +1177,34 @@ static bool is_end_zone_blkaddr(struct f2fs_sb_info *sbi, block_t blkaddr) > f2fs_blkz_is_seq(sbi, devi, blkaddr) && > (blkaddr % sbi->blocks_per_blkz == sbi->blocks_per_blkz - 1); > } > + > +static void f2fs_wait_zone_io_completion(struct f2fs_sb_info *sbi, > + struct f2fs_bio_info *io, enum page_type btype) > +{ > + if (f2fs_sb_has_blkzoned(sbi) && btype < META && io->zone_pending_bio) { > + wait_for_completion_io(&io->zone_wait); > + bio_put(io->zone_pending_bio); > + io->zone_pending_bio = NULL; > + io->bi_private = NULL; > + } > +} > + > +static void f2fs_submit_zone_io(struct f2fs_sb_info *sbi, > + struct f2fs_io_info *fio, struct f2fs_bio_info *io, > + enum page_type btype) > +{ > + if (f2fs_sb_has_blkzoned(sbi) && btype < META && > + is_end_zone_blkaddr(sbi, fio->new_blkaddr)) { > + bio_get(io->bio); > + reinit_completion(&io->zone_wait); > + io->bi_private = io->bio->bi_private; > + io->bio->bi_private = io; > + io->bio->bi_end_io = f2fs_zone_write_end_io; > + io->zone_pending_bio = io->bio; > + __submit_merged_bio(io); > + } > + > +} > #endif > > void f2fs_submit_page_write(struct f2fs_io_info *fio) > @@ -1079,14 +1221,8 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio) > f2fs_down_write_trace(&io->io_rwsem, &lc); > next: > #ifdef CONFIG_BLK_DEV_ZONED > - if (f2fs_sb_has_blkzoned(sbi) && btype < META && io->zone_pending_bio) { > - wait_for_completion_io(&io->zone_wait); > - bio_put(io->zone_pending_bio); > - io->zone_pending_bio = NULL; > - io->bi_private = NULL; > - } > + f2fs_wait_zone_io_completion(sbi, io, btype); > #endif > - > if (fio->in_list) { > spin_lock(&io->io_lock); > if (list_empty(&io->io_list)) { > @@ -1141,16 +1277,109 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio) > > trace_f2fs_submit_folio_write(fio->folio, fio); > #ifdef CONFIG_BLK_DEV_ZONED > - if (f2fs_sb_has_blkzoned(sbi) && btype < META && > - is_end_zone_blkaddr(sbi, fio->new_blkaddr)) { > - bio_get(io->bio); > - reinit_completion(&io->zone_wait); > - io->bi_private = io->bio->bi_private; > - io->bio->bi_private = io; > - io->bio->bi_end_io = f2fs_zone_write_end_io; > - io->zone_pending_bio = io->bio; > + f2fs_submit_zone_io(sbi, fio, io, btype); > +#endif > + > + if (fio->in_list) > + goto next; > +out: > + if (is_sbi_flag_set(sbi, SBI_IS_SHUTDOWN) || > + !f2fs_is_checkpoint_ready(sbi)) > __submit_merged_bio(io); > + f2fs_up_write_trace(&io->io_rwsem, &lc); > +} > + > +static void f2fs_bio_add_cache(struct f2fs_io_info *fio, struct bio *bio) > +{ > + struct f2fs_bio *fbio = F2FS_BIO(bio); > + struct f2fs_cached_block *head = fbio->entry; > + struct f2fs_cached_block *new = fio->cache_entry; > + > + new->next_entry = head; > + fbio->entry = new; > +} > + > +int f2fs_submit_cache_read(struct f2fs_io_info *fio) > +{ > + struct f2fs_sb_info *sbi = fio->sbi; > + struct f2fs_cached_block *entry = fio->cache_entry; > + struct bio *bio; > + enum count_type io_type = IS_META_CACHE(entry->cache) ? > + F2FS_RD_META : F2FS_RD_NODE; > + > + if (!f2fs_is_valid_blkaddr(fio->sbi, fio->new_blkaddr, > + fio->is_por ? META_POR : (__is_meta_io(fio) ? > + META_GENERIC : DATA_GENERIC_ENHANCE))) > + return -EFSCORRUPTED; > + > + bio = __bio_alloc(fio, 1); > + > + bio_add_virt_nofail(bio, cache_address(entry), sbi->blocksize); > + f2fs_bio_add_cache(fio, bio); > + inc_page_count(sbi, io_type); > + > + f2fs_submit_read_bio(sbi, bio, fio->type); > + return 0; > +} > + > +void f2fs_submit_cache_write(struct f2fs_io_info *fio) > +{ > + struct f2fs_sb_info *sbi = fio->sbi; > + enum page_type btype = PAGE_TYPE_OF_BIO(fio->type); > + struct f2fs_bio_info *io = sbi->write_io[btype] + fio->temp; > + struct f2fs_lock_context lc; > + struct folio *folio; > + > + f2fs_bug_on(sbi, is_read_io(fio->op)); > + > + f2fs_down_write_trace(&io->io_rwsem, &lc); > +next: > +#ifdef CONFIG_BLK_DEV_ZONED > + f2fs_wait_zone_io_completion(sbi, io, btype); > +#endif > + if (fio->in_list) { > + spin_lock(&io->io_lock); > + if (list_empty(&io->io_list)) { > + spin_unlock(&io->io_lock); > + goto out; > + } > + fio = list_first_entry(&io->io_list, > + struct f2fs_io_info, list); > + list_del(&fio->list); > + spin_unlock(&io->io_lock); > } > + > + verify_fio_blkaddr(fio); > + > + fio->submitted = 1; > + inc_page_count(sbi, F2FS_WB_CP_DATA); > + > + if (io->bio && > + (!io_is_mergeable(sbi, io->bio, io, fio, io->last_block_in_bio, > + fio->new_blkaddr))) > + __submit_merged_bio(io); > +alloc_new: > + if (io->bio == NULL) { > + io->bio = __bio_alloc(fio, BIO_MAX_VECS); > + io->fio = *fio; > + } > + > + folio = cache_folio(fio->cache_entry); > + > + if (!bio_add_folio(io->bio, folio, sbi->blocksize, > + offset_in_folio(folio, cache_address(fio->cache_entry)))) { > + f2fs_bug_on(sbi, !F2FS_BIO(io->bio)->entry); > + > + __submit_merged_bio(io); > + goto alloc_new; > + } > + > + f2fs_bio_add_cache(fio, io->bio); > + > + io->last_block_in_bio = fio->new_blkaddr; > + > +#ifdef CONFIG_BLK_DEV_ZONED > + f2fs_submit_zone_io(sbi, fio, io, btype); > #endif > if (fio->in_list) > goto next; > diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h > index 1b96d8718c5c..8413983ea9d5 100644 > --- a/fs/f2fs/f2fs.h > +++ b/fs/f2fs/f2fs.h > @@ -221,6 +221,8 @@ struct f2fs_rwsem { > #endif > }; > > +#include "cache.h" > + > struct f2fs_mount_info { > unsigned long long opt; > block_t root_reserved_blocks; /* root reserved blocks */ > @@ -1370,8 +1372,10 @@ struct f2fs_io_info { > unsigned int is_por:1; /* indicate IO is from recovery or not */ > unsigned int encrypted:1; /* indicate file is encrypted */ > unsigned int meta_gc:1; /* require meta inode GC */ > + unsigned int is_cache:1; /* indicate IO is from internal cache */ > enum iostat_type io_type; /* io type */ > struct writeback_control *io_wbc; /* writeback control */ > + struct f2fs_cached_block *cache_entry; > struct bio **bio; /* bio for ipu */ > sector_t *last_block; /* last block number in bio */ > }; > @@ -1781,6 +1785,12 @@ struct f2fs_gc_kthread { > unsigned int boost_gc_greedy; > }; > > +struct f2fs_bio { > + struct work_struct work; > + struct f2fs_cached_block *entry; > + struct bio bio; > +}; > + > struct f2fs_sb_info { > struct super_block *sb; /* pointer to VFS super block */ > struct proc_dir_entry *s_proc; /* proc entry */ > @@ -2323,6 +2333,16 @@ static inline bool is_node_folio(struct folio *folio) > return folio->mapping == NODE_MAPPING(F2FS_F_SB(folio)); > } > > +static inline struct f2fs_bio *F2FS_BIO(struct bio *bio) > +{ > + return container_of(bio, struct f2fs_bio, bio); > +} > + > +static inline bool f2fs_is_cache_bio(struct bio *bio) > +{ > + return F2FS_BIO(bio)->entry != NULL; > +} > + > static inline bool is_sbi_flag_set(struct f2fs_sb_info *sbi, unsigned int type) > { > return test_bit(type, &sbi->s_flag); > @@ -4237,6 +4257,8 @@ void f2fs_submit_merged_write_cond(struct f2fs_sb_info *sbi, > nid_t ino, enum page_type type); > void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, > struct folio *folio, enum page_type type); > +bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, > + enum page_type type); > void f2fs_submit_merged_ipu_write(struct f2fs_sb_info *sbi, > struct bio **bio, struct folio *folio); > void f2fs_submit_all_merged_ipu_writes(struct f2fs_sb_info *sbi); > @@ -4244,6 +4266,8 @@ void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi); > int f2fs_submit_page_bio(struct f2fs_io_info *fio); > int f2fs_merge_page_bio(struct f2fs_io_info *fio); > void f2fs_submit_page_write(struct f2fs_io_info *fio); > +int f2fs_submit_cache_read(struct f2fs_io_info *fio); > +void f2fs_submit_cache_write(struct f2fs_io_info *fio); > struct block_device *f2fs_target_device(struct f2fs_sb_info *sbi, > block_t blk_addr, sector_t *sector); > int f2fs_target_device_index(struct f2fs_sb_info *sbi, block_t blkaddr); > diff --git a/fs/f2fs/iostat.h b/fs/f2fs/iostat.h > index 2025225b5bed..61c6bc8e3119 100644 > --- a/fs/f2fs/iostat.h > +++ b/fs/f2fs/iostat.h > @@ -60,6 +60,13 @@ static inline struct bio_post_read_ctx *get_post_read_ctx(struct bio *bio) > return iostat_ctx->post_read_ctx; > } > > +static inline void iostat_set_post_read_ctx(struct bio *bio, void *ctx) > +{ > + struct bio_iostat_ctx *iostat_ctx = bio->bi_private; > + > + iostat_ctx->post_read_ctx = ctx; > +} > + > extern void iostat_update_and_unbind_ctx(struct bio *bio); > extern void iostat_alloc_and_bind_ctx(struct f2fs_sb_info *sbi, > struct bio *bio, struct bio_post_read_ctx *ctx); > @@ -81,6 +88,10 @@ static inline struct bio_post_read_ctx *get_post_read_ctx(struct bio *bio) > { > return bio->bi_private; > } > +static inline void iostat_set_post_read_ctx(struct bio *bio, void *ctx) > +{ > + bio->bi_private = ctx; > +} > static inline int f2fs_init_iostat_processing(void) { return 0; } > static inline void f2fs_destroy_iostat_processing(void) {} > static inline int f2fs_init_iostat(struct f2fs_sb_info *sbi) { return 0; } > -- > 2.49.0 > > > > _______________________________________________ > Linux-f2fs-devel mailing list > Linux-f2fs-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [f2fs-dev] [PATCH v1 01/12] f2fs: cache: implement metadata cache @ 2026-08-20 5:08 ` Jaegeuk Kim via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Jaegeuk Kim via Linux-f2fs-devel @ 2026-08-20 5:08 UTC (permalink / raw) To: Chao Yu; +Cc: linux-kernel, linux-f2fs-devel On 08/20, Chao Yu via Linux-f2fs-devel wrote: > This patch introduces the core metadata block caching infrastructure to > manage f2fs metadata independently of the page cache. > > It implements: > - core cache APIs: get, create, put, drop, backed by a radix tree and > a single global LRU list. > - support multiple status of cached block: LOCKED, UPTODATE, DIRTY, > WRITEBACK, INLINE. > - internal bio based read/write helpers with adjacent block vector merging. > > Signed-off-by: Chao Yu <chao@kernel.org> > --- > fs/f2fs/Makefile | 2 +- > fs/f2fs/cache.c | 531 +++++++++++++++++++++++++++++++++++++++++++++++ > fs/f2fs/cache.h | 179 ++++++++++++++++ > fs/f2fs/data.c | 283 ++++++++++++++++++++++--- > fs/f2fs/f2fs.h | 24 +++ > fs/f2fs/iostat.h | 11 + > 6 files changed, 1002 insertions(+), 28 deletions(-) > create mode 100644 fs/f2fs/cache.c > create mode 100644 fs/f2fs/cache.h > > diff --git a/fs/f2fs/Makefile b/fs/f2fs/Makefile > index 8a7322d229e4..fbf49c30b066 100644 > --- a/fs/f2fs/Makefile > +++ b/fs/f2fs/Makefile > @@ -3,7 +3,7 @@ obj-$(CONFIG_F2FS_FS) += f2fs.o > > f2fs-y := dir.o file.o inode.o namei.o hash.o super.o inline.o > f2fs-y += checkpoint.o gc.o data.o node.o segment.o recovery.o > -f2fs-y += shrinker.o extent_cache.o sysfs.o > +f2fs-y += shrinker.o extent_cache.o sysfs.o cache.o > f2fs-$(CONFIG_F2FS_STAT_FS) += debug.o > f2fs-$(CONFIG_F2FS_FS_XATTR) += xattr.o > f2fs-$(CONFIG_F2FS_FS_POSIX_ACL) += acl.o > diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c > new file mode 100644 > index 000000000000..08bc658166f7 > --- /dev/null > +++ b/fs/f2fs/cache.c > @@ -0,0 +1,531 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * Copyright (c) 2026 Google LLC > + * Author: Chao Yu <chaseyu@google.com> > + */ > +#include "linux/spinlock.h" #include <linux/spinlock.h>? May need to Move below? > +#include <linux/fs.h> > +#include <linux/f2fs_fs.h> > +#include <linux/radix-tree.h> > +#include <linux/slab.h> > +#include <linux/list.h> > +#include <linux/pagemap.h> > +#include <linux/kthread.h> > +#include <linux/freezer.h> > +#include <linux/delay.h> > +#include "f2fs.h" > +#include "cache.h" > +#include "node.h" > +#include "segment.h" > + > +void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, > + enum page_type type) > +{ > + /* in case the entry was truncated or on-going shrink */ > + if (!entry->cache) > + return; > + > + if (!f2fs_cache_test_writeback(entry)) > + return; > + > + /* submit cached bio */ > + f2fs_submit_merged_write_cache(entry, type); > + > + wait_on_bit_io(&entry->state, F2FS_BLOCK_WRITEBACK, > + TASK_UNINTERRUPTIBLE); > +} > + > +void f2fs_cache_wait_writeback(struct f2fs_cached_block *entry) > +{ > + /* in case the entry was truncated or on-going shrink */ > + if (!entry->cache) > + return; > + > + f2fs_cache_wait_writeback_cond(entry, > + IS_META_CACHE(entry->cache) ? META : NODE); > +} > + > +static void f2fs_cache_update_tag(struct f2fs_cached_block *entry, unsigned int src, > + unsigned int dst) f2fs_cache_update_tag(entry, clear_from, set_to); > +{ > + struct f2fs_cached_block_list *cache = entry->cache; > + unsigned long flags; > + > + spin_lock_irqsave(&cache->tree_lock, flags); > + if (src) if (clear_fom != F2FS_CACHE_TAG_NONE) > + radix_tree_tag_clear(&cache->root, entry->index, src); > + if (dst) if (set_to != F2FS_CACHE_TAG_NONE) > + radix_tree_tag_set(&cache->root, entry->index, dst); > + spin_unlock_irqrestore(&cache->tree_lock, flags); > +} > + > +bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry) > +{ > + struct f2fs_cached_block_list *cache = entry->cache; > + > + if (!f2fs_cache_test_uptodate(entry)) > + f2fs_cache_set_uptodate(entry); f2fs_cache_set_uptodate(entry); > + > +#ifdef CONFIG_F2FS_CHECK_FS > + if (cache->type == F2FS_NODE_CACHE && IS_INODE(cache_folio(entry))) > + f2fs_inode_chksum_set(cache->sbi, cache_folio(entry)); > +#endif > + > + if (f2fs_cache_test_dirty(entry)) > + return false; > + > + if (!f2fs_cache_test_and_set_dirty(entry)) { > + enum count_type type = IS_META_CACHE(cache) ? > + F2FS_DIRTY_META : F2FS_DIRTY_NODES; > + > + f2fs_cache_update_tag(entry, 0, F2FS_CACHE_TAG_DIRTY); f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_NONE, F2FS_CACHE_TAG_DIRTY); > + inc_page_count(cache->sbi, type); > + return true; > + } > + > + return false; > +} > + > +bool f2fs_clear_cache_dirty(struct f2fs_cached_block *entry) > +{ > + if (!f2fs_cache_test_dirty(entry)) > + return false; > + > + f2fs_cache_clear_dirty(entry); > + return true; > +} > + > +static void __f2fs_drop_cache_dirty(struct f2fs_cached_block *entry, bool force) static void __drop_cache_dirty(struct f2fs_cached_block *entry, bool force) > +{ > + > + struct f2fs_cached_block_list *cache = entry->cache; > + enum count_type type = IS_META_CACHE(cache) ? > + F2FS_DIRTY_META : F2FS_DIRTY_NODES; > + > + f2fs_cache_clear_uptodate(entry); > + > + if (!force && !f2fs_clear_cache_dirty(entry)) > + return; > + > + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_DIRTY, 0); > + dec_page_count(cache->sbi, type); > +} > + > +void f2fs_drop_cache_dirty(struct f2fs_cached_block *entry) > +{ > + __f2fs_drop_cache_dirty(entry, false); > +} > + > +void f2fs_force_clear_cache_dirty(struct f2fs_cached_block *entry) > +{ > + __f2fs_drop_cache_dirty(entry, true); > +} > + > +void f2fs_start_cache_writeback(struct f2fs_cached_block *entry) > +{ > + f2fs_cache_set_writeback(entry); > + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_DIRTY, > + F2FS_CACHE_TAG_WRITEBACK); > +} > + > +void f2fs_end_cache_writeback(struct f2fs_cached_block *entry) > +{ > + /* > + * should call f2fs_cache_update_tag() before clearing writeback bit, > + * in case f2fs_truncate_cache() set entry->cache to NULL. > + */ > + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_WRITEBACK, 0); > + clear_and_wake_up_bit(F2FS_BLOCK_WRITEBACK, &entry->state); > +} > + > +static int f2fs_cache_refcount(struct f2fs_cached_block *entry) > +{ > + return atomic_read(&entry->refcount); > +} > + > +static void __f2fs_free_cache(struct f2fs_cached_block *entry) > +{ > + kfree(entry->data); > + kfree(entry); > +} > + > +static void f2fs_free_cache(struct f2fs_cached_block *entry) > +{ > + WARN_ON_ONCE(!list_empty(&entry->list)); > + WARN_ON_ONCE(f2fs_cache_refcount(entry)); > + __f2fs_free_cache(entry); > +} > + > +void f2fs_cache_get(struct f2fs_cached_block *entry) > +{ > + atomic_inc(&entry->refcount); > +} > + > +static bool f2fs_cache_put(struct f2fs_cached_block *entry) > +{ > + WARN_ON_ONCE(!f2fs_cache_refcount(entry)); > + if (atomic_dec_and_test(&entry->refcount)) { > + f2fs_free_cache(entry); > + return true; > + } > + return false; > +} > + > +static struct f2fs_cached_block *f2fs_create_cache( > + struct f2fs_cached_block_list *cache, > + unsigned long index, bool nofail) > +{ > + struct f2fs_cached_block *entry; > + unsigned int flags = GFP_NOFS; > + > + if (nofail) > + flags |= __GFP_NOFAIL; > + > + entry = kzalloc_obj(*entry, flags); > + if (!entry) > + return ERR_PTR(-ENOMEM); > + > + entry->data = kzalloc(cache->sbi->blocksize, flags); We don't need kzalloc() since we have an uptodate flag. > + if (!entry->data) { > + kfree(entry); > + return ERR_PTR(-ENOMEM); > + } > + > + entry->index = index; > + > + atomic_set(&entry->refcount, 0); > + entry->next_entry = NULL; > + INIT_LIST_HEAD(&entry->list); > + > + entry->cache = cache; > + > + return entry; > +} > + > +static struct f2fs_cached_block *f2fs_insert_cache( > + struct f2fs_cached_block_list *cache, > + unsigned long index, > + struct f2fs_cached_block *new) > +{ > + struct f2fs_cached_block *e; > + int ret; > + unsigned long flags; > + > + ret = radix_tree_preload(GFP_NOFS | __GFP_NOFAIL); > + f2fs_bug_on(cache->sbi, ret); > + > + spin_lock(&cache->list_lock); > + spin_lock_irqsave(&cache->tree_lock, flags); > + e = radix_tree_lookup(&cache->root, index); > + if (!e) { > + e = new; > + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(e)); > + > + ret = radix_tree_insert(&cache->root, index, e); > + f2fs_bug_on(cache->sbi, ret); > + > + /* radix tree referenced cache entry */ > + f2fs_cache_get(e); > + f2fs_bug_on(cache->sbi, !list_empty(&e->list)); > + list_add_tail(&e->list, &cache->lru_list); > + cache->num_entries++; > + } > + f2fs_cache_get(e); > + spin_unlock_irqrestore(&cache->tree_lock, flags); > + spin_unlock(&cache->list_lock); > + radix_tree_preload_end(); > + > + if (new != e) { > + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(new)); > + __f2fs_free_cache(new); > + } > + > + return e; > +} > + > +struct f2fs_cached_block *f2fs_find_cache( > + struct f2fs_cached_block_list *cache, > + unsigned long index) > +{ > + struct f2fs_cached_block *entry; > + unsigned long flags; > + > + spin_lock(&cache->list_lock); > + spin_lock_irqsave(&cache->tree_lock, flags); > + entry = radix_tree_lookup(&cache->root, index); > + if (entry) { > + f2fs_bug_on(cache->sbi, !f2fs_cache_refcount(entry)); > + f2fs_cache_get(entry); > + if (!list_empty(&entry->list)) > + list_move_tail(&entry->list, &cache->lru_list); > + > + } else { > + entry = ERR_PTR(-ENOENT); > + } > + spin_unlock_irqrestore(&cache->tree_lock, flags); > + spin_unlock(&cache->list_lock); > + > + return entry; > +} > + > +struct f2fs_cached_block *f2fs_grab_cache( > + struct f2fs_cached_block_list *cache, > + unsigned long index, int flags) > + > +{ > + struct f2fs_cached_block *entry, *new; > + bool create = flags & F2FS_CACHE_CREATE; > + bool nofail = flags & F2FS_CACHE_NOFAIL; > + bool lock = flags & F2FS_CACHE_LOCK; > + > +repeat: > + entry = f2fs_find_cache(cache, index); > + if (!IS_ERR(entry)) > + goto found; > + > + if (!create) > + return ERR_PTR(-ENOENT); > + > + new = f2fs_create_cache(cache, index, nofail); > + if (IS_ERR(new)) > + return new; > + > + entry = f2fs_insert_cache(cache, index, new); > +found: > + if (lock) { > + f2fs_lock_cache(entry); > + /* has been truncated */ > + if (entry->cache != cache) { > + f2fs_put_cache(entry, true); > + goto repeat; > + } > + } > + return entry; > +} > + > +bool f2fs_trylock_cache(struct f2fs_cached_block *entry) > +{ > + return !test_and_set_bit(F2FS_BLOCK_LOCKED, &entry->state); > +} > + > +void f2fs_lock_cache(struct f2fs_cached_block *entry) > +{ > + wait_on_bit_lock(&entry->state, F2FS_BLOCK_LOCKED, > + TASK_UNINTERRUPTIBLE); > +} > + > +void f2fs_unlock_cache(struct f2fs_cached_block *entry) > +{ > + clear_and_wake_up_bit(F2FS_BLOCK_LOCKED, &entry->state); > +} > + > +bool f2fs_put_cache(struct f2fs_cached_block *entry, bool unlock) > +{ > + if (IS_ERR_OR_NULL(entry)) > + return false; > + if (unlock) > + f2fs_unlock_cache(entry); > + return f2fs_cache_put(entry); > +} > + > +unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache, > + struct f2fs_cached_block **entries, > + pgoff_t *first_index, unsigned int max_nr, > + int tag) > +{ > + unsigned long flags; > + int nr, i; > + > + spin_lock_irqsave(&cache->tree_lock, flags); > + nr = radix_tree_gang_lookup_tag(&cache->root, (void **)entries, > + *first_index, max_nr, tag); > + if (!nr) > + goto out; > + > + for (i = 0; i < nr; i++) > + f2fs_cache_get(entries[i]); > + *first_index = entries[nr - 1]->index + 1; > +out: > + spin_unlock_irqrestore(&cache->tree_lock, flags); > + return nr; > +} > + > +void f2fs_cache_gang_release(struct f2fs_cached_block **entries, > + unsigned int nr_entries) > +{ > + int i; > + > + for (i = 0; i < nr_entries; i++) > + f2fs_put_cache(entries[i], false); > +} > + > +void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache) > +{ > + unsigned long index = 0; > + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; > + int nr, i; > + > +next: > + nr = f2fs_cache_gang_lookup_tag(cache, entries, &index, > + F2FS_ONSTACK_CACHES, F2FS_CACHE_TAG_WRITEBACK); > + if (!nr) > + return; > + > + for (i = 0; i < nr; i++) > + f2fs_cache_wait_writeback(entries[i]); > + f2fs_cache_gang_release(entries, nr); > + goto next; > +} > + > +static void f2fs_do_truncate_cache(struct f2fs_cached_block *entry, > + bool drop_dirty) > +{ > + struct f2fs_cached_block_list *cache = entry->cache; > + unsigned long flags; > + > + if (!drop_dirty && > + (f2fs_cache_test_dirty(entry) || > + f2fs_cache_test_writeback(entry))) > + return; > + > + f2fs_cache_wait_writeback(entry); > + f2fs_drop_cache_dirty(entry); > + > + spin_lock(&cache->list_lock); > + spin_lock_irqsave(&cache->tree_lock, flags); > + > + f2fs_bug_on(cache->sbi, !entry->cache); > + if (!radix_tree_delete(&cache->root, entry->index)) > + f2fs_bug_on(cache->sbi, !entry->cache); > + > + entry->cache = NULL; > + cache->num_entries--; > + > + atomic_dec(&entry->refcount); > + f2fs_bug_on(cache->sbi, !f2fs_cache_refcount(entry)); > + > + f2fs_bug_on(cache->sbi, list_empty(&entry->list)); > + list_del_init(&entry->list); > + > + spin_unlock_irqrestore(&cache->tree_lock, flags); > + spin_unlock(&cache->list_lock); > +} > + > +static void f2fs_truncate_cache(struct f2fs_cached_block *entry, > + bool drop_dirty) > +{ > + f2fs_lock_cache(entry); > + if (entry->cache) > + f2fs_do_truncate_cache(entry, drop_dirty); > + f2fs_unlock_cache(entry); > +} > + > +static void f2fs_drop_cache(struct f2fs_cached_block_list *cache, > + block_t blkaddr, bool drop_dirty) > +{ > + struct f2fs_cached_block *entry; > + > + entry = f2fs_find_cache(cache, blkaddr); > + if (IS_ERR(entry)) > + return; > + > + f2fs_truncate_cache(entry, drop_dirty); > + f2fs_put_cache(entry, false); > +} > + > +void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache, > + unsigned long start, unsigned long len, bool drop_dirty) > +{ > + unsigned long index = start; > + unsigned long end = (ULONG_MAX - start < len) ? > + ULONG_MAX : (start + len); > + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; > + unsigned long flags; > + int nr, i; > + > + if (len == 1) > + return f2fs_drop_cache(cache, index, drop_dirty); > + > +next: > + spin_lock_irqsave(&cache->tree_lock, flags); > + nr = radix_tree_gang_lookup(&cache->root, (void **)entries, index, > + min((unsigned long)F2FS_ONSTACK_CACHES, end - index)); > + if (!nr) > + goto out_unlock; > + > + for (i = 0; i < nr; i++) { > + struct f2fs_cached_block *entry = entries[i]; > + > + if (entry->index >= end) { > + nr = i; > + break; > + } > + f2fs_cache_get(entry); > + } > +out_unlock: > + spin_unlock_irqrestore(&cache->tree_lock, flags); > + if (!nr) > + return; > + > + for (i = 0; i < nr; i++) { > + struct f2fs_cached_block *entry = entries[i]; > + > + index = entry->index + 1; > + > + f2fs_truncate_cache(entry, drop_dirty); > + } > + f2fs_cache_gang_release(entries, nr); > + > + if (index < end) > + goto next; > +} > + > +int f2fs_init_cache(struct f2fs_sb_info *sbi, > + struct f2fs_cached_block_list *cache, > + enum f2fs_cache_type type) > +{ > + cache->sbi = sbi; > + cache->type = type; > + INIT_RADIX_TREE(&cache->root, GFP_ATOMIC); > + spin_lock_init(&cache->tree_lock); > + spin_lock_init(&cache->list_lock); > + INIT_LIST_HEAD(&cache->lru_list); > + cache->num_entries = 0; > + > + return 0; > +} > + > +void f2fs_destroy_cache(struct f2fs_cached_block_list *cache) > +{ > + struct list_head *head = &cache->lru_list; > + struct f2fs_cached_block *entry; > + unsigned long flags; > + > + f2fs_cache_wait_on_all_writeback(cache); > +next: > + spin_lock(&cache->list_lock); > + if (list_empty(head)) { > + spin_unlock(&cache->list_lock); > + return; > + } > + entry = list_first_entry(head, struct f2fs_cached_block, list); > + > + spin_lock_irqsave(&cache->tree_lock, flags); > + radix_tree_delete(&cache->root, entry->index); > + cache->num_entries--; > + list_del_init(&entry->list); > + spin_unlock_irqrestore(&cache->tree_lock, flags); > + > + spin_unlock(&cache->list_lock); > + > + /* wait on read cache IO */ > + f2fs_lock_cache(entry); > + /* wait on write cache IO */ > + f2fs_cache_wait_writeback(entry); > + f2fs_bug_on(cache->sbi, f2fs_cache_test_dirty(entry)); > + f2fs_bug_on(cache->sbi, f2fs_cache_test_writeback(entry)); > + f2fs_bug_on(cache->sbi, !list_empty(&entry->list)); > + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(entry) != 1); > + f2fs_put_cache(entry, true); > + goto next; > +} > diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h > new file mode 100644 > index 000000000000..686a974008ae > --- /dev/null > +++ b/fs/f2fs/cache.h > @@ -0,0 +1,179 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * Copyright (c) 2026 Google LLC > + * Author: Chao Yu <chaseyu@google.com> > + */ > +#ifndef _LINUX_F2FS_CACHE_H > +#define _LINUX_F2FS_CACHE_H > + > +#include <linux/pagemap.h> > +#include <linux/mm.h> > +#include <linux/list.h> > +#include <linux/radix-tree.h> > +#include <linux/spinlock.h> > +#include <linux/wait.h> > +#include <linux/types.h> > + > +struct f2fs_rwsem; > +struct f2fs_io_info; > +enum page_type; > + > +/* Represents a single cached block (meta, node or compress) */ > +struct f2fs_cached_block { > + struct list_head list; /* LRU list head */ > + struct f2fs_cached_block_list *cache; /* parent cache list */ > + union { > + /* chain for merged BIO */ > + struct f2fs_cached_block *next_entry; > + nid_t ino; /* inode number for compress cache */ > + }; > + unsigned long index; /* key in radix tree, (meta/compress: pba, node: nid) */ > + unsigned long state; /* cache entry state (e.g., Dirty, UpToDate) */ > + void *data; /* blocksize-aligned memory (4KB or 16KB) */ > + atomic_t refcount; /* reference count */ > +}; > + > +struct f2fs_sb_info; > + > +enum f2fs_cache_type { > + F2FS_META_CACHE, > + F2FS_NODE_CACHE, > +}; > + > +/* Main cache control structure (per sb_info) */ > +struct f2fs_cached_block_list { > + struct f2fs_sb_info *sbi; /* Pointer to f2fs_sb_info */ > + struct radix_tree_root root; /* Radix tree for cache lookup */ > + spinlock_t tree_lock; /* Lock for radix tree */ > + struct list_head lru_list; /* Single global LRU list */ > + spinlock_t list_lock; /* Lock for LRU list */ > + enum f2fs_cache_type type; /* Cache type (Node or Meta) */ > + unsigned long num_entries; /* Current number of entries */ > +}; > + > +#define IS_META_CACHE(cache) (cache->type == F2FS_META_CACHE) > + > +/* Flags for f2fs_cached_block state */ > +enum f2fs_cached_state { > + F2FS_BLOCK_LOCKED, /* cache entry is locked */ > + F2FS_BLOCK_UPTODATE, /* cache data is valid */ > + F2FS_BLOCK_DIRTY, /* cache data is dirty, need to writeback the data */ > + F2FS_BLOCK_WRITEBACK, /* cache data is writeback state */ > + F2FS_BLOCK_INLINE_DATA, /* indicate inline data */ > +}; > + > +enum { > + __F2FS_CACHE_CREATE, /* create the cache if there is no cache entry */ > + __F2FS_CACHE_LOCK, /* get and lock the cache entry */ > + __F2FS_CACHE_NOFAIL, /* do not allow failure */ > +}; > + > +enum f2fs_cache_request_flag { > + F2FS_CACHE_CREATE = 1 << __F2FS_CACHE_CREATE, > + F2FS_CACHE_LOCK = 1 << __F2FS_CACHE_LOCK, > + F2FS_CACHE_NOFAIL = 1 << __F2FS_CACHE_NOFAIL, > +}; > + > +#define F2FS_CACHE_LOCK_CREATE (F2FS_CACHE_LOCK | F2FS_CACHE_CREATE) > + > +#define F2FS_ONSTACK_CACHES (32) > + > +#define F2FS_CACHE_FLAG_TEST_FUNC(name, flagname) \ > +static inline bool f2fs_cache_test_##name( \ > + const struct f2fs_cached_block *entry) \ > +{ \ > + return test_bit(F2FS_BLOCK_##flagname, &entry->state); \ > +} \ > + > +#define F2FS_CACHE_FLAG_SET_FUNC(name, flagname) \ > +static inline void f2fs_cache_set_##name( \ > + struct f2fs_cached_block *entry) \ > +{ \ > + set_bit(F2FS_BLOCK_##flagname, &entry->state); \ > +} \ > + > +#define F2FS_CACHE_FLAG_CLEAR_FUNC(name, flagname) \ > +static inline void f2fs_cache_clear_##name( \ > + struct f2fs_cached_block *entry) \ > +{ \ > + clear_bit(F2FS_BLOCK_##flagname, &entry->state); \ > +} \ > + > +#define F2FS_CACHE_FLAG_TEST_AND_SET_FUNC(name, flagname) \ > +static inline bool f2fs_cache_test_and_set_##name( \ > + struct f2fs_cached_block *entry) \ > +{ \ > + return test_and_set_bit(F2FS_BLOCK_##flagname, &entry->state); \ > +} \ > + > +F2FS_CACHE_FLAG_TEST_FUNC(locked, LOCKED); > +F2FS_CACHE_FLAG_SET_FUNC(locked, LOCKED); > +F2FS_CACHE_FLAG_CLEAR_FUNC(locked, LOCKED); > + > +F2FS_CACHE_FLAG_TEST_FUNC(uptodate, UPTODATE); > +F2FS_CACHE_FLAG_SET_FUNC(uptodate, UPTODATE); > +F2FS_CACHE_FLAG_CLEAR_FUNC(uptodate, UPTODATE); > + > +F2FS_CACHE_FLAG_TEST_FUNC(dirty, DIRTY); > +F2FS_CACHE_FLAG_SET_FUNC(dirty, DIRTY); > +F2FS_CACHE_FLAG_CLEAR_FUNC(dirty, DIRTY); > +F2FS_CACHE_FLAG_TEST_AND_SET_FUNC(dirty, DIRTY); > + > +F2FS_CACHE_FLAG_TEST_FUNC(writeback, WRITEBACK); > +F2FS_CACHE_FLAG_SET_FUNC(writeback, WRITEBACK); > +F2FS_CACHE_FLAG_CLEAR_FUNC(writeback, WRITEBACK); > + > +F2FS_CACHE_FLAG_TEST_FUNC(inline, INLINE_DATA); > +F2FS_CACHE_FLAG_SET_FUNC(inline, INLINE_DATA); > +F2FS_CACHE_FLAG_CLEAR_FUNC(inline, INLINE_DATA); > + > +static inline void *cache_address(const struct f2fs_cached_block *entry) > +{ > + return entry->data; > +} > + > +#define CACHED_NODE(entry) ((struct f2fs_node *)(cache_address(entry))) > + > +static inline struct folio *cache_folio(const struct f2fs_cached_block *entry) > +{ > + return virt_to_folio(entry->data); > +} > + > +int f2fs_init_cache(struct f2fs_sb_info *sbi, > + struct f2fs_cached_block_list *cache, > + enum f2fs_cache_type type); > +void f2fs_destroy_cache(struct f2fs_cached_block_list *cache); > +void f2fs_cache_get(struct f2fs_cached_block *entry); > +struct f2fs_cached_block *f2fs_find_cache( > + struct f2fs_cached_block_list *cache, > + unsigned long index); > +#define F2FS_CACHE_TAG_NONE 0 > +#define F2FS_CACHE_TAG_DIRTY 1 > +#define F2FS_CACHE_TAG_WRITEBACK 2 > + > +bool f2fs_trylock_cache(struct f2fs_cached_block *entry); > +void f2fs_lock_cache(struct f2fs_cached_block *entry); > +void f2fs_unlock_cache(struct f2fs_cached_block *entry); > +bool f2fs_put_cache(struct f2fs_cached_block *entry, bool unlock); > +bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry); > +bool f2fs_clear_cache_dirty(struct f2fs_cached_block *entry); > +void f2fs_drop_cache_dirty(struct f2fs_cached_block *entry); > +void f2fs_force_clear_cache_dirty(struct f2fs_cached_block *entry); > +void f2fs_start_cache_writeback(struct f2fs_cached_block *entry); > +void f2fs_end_cache_writeback(struct f2fs_cached_block *entry); > +unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache, > + struct f2fs_cached_block **results, pgoff_t *first_index, > + unsigned int max_items, int tag); > +void f2fs_cache_gang_release(struct f2fs_cached_block **entries, > + unsigned int nr_entries); > +int f2fs_writeback_cache(struct f2fs_cached_block_list *cache, bool sync); > +void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache); > +void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, > + enum page_type type); > +void f2fs_cache_wait_writeback(struct f2fs_cached_block *entry); > +struct f2fs_cached_block *f2fs_grab_cache(struct f2fs_cached_block_list *cache, > + unsigned long index, int flags); > +void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache, > + unsigned long start, unsigned long len, bool drop_dirty); > + > +#endif /* _LINUX_F2FS_CACHE_H */ > diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c > index 6ae0eb37d20f..09474790035b 100644 > --- a/fs/f2fs/data.c > +++ b/fs/f2fs/data.c > @@ -41,11 +41,6 @@ struct f2fs_folio_state { > unsigned int read_pages_pending; > }; > > -struct f2fs_bio { > - struct work_struct work; > - struct bio bio; > -}; > - > #define F2FS_BIO_POOL_SIZE NR_CURSEG_TYPE > > int __init f2fs_init_bioset(void) > @@ -69,7 +64,6 @@ bool f2fs_is_cp_guaranteed(const struct folio *folio) > return folio_test_f2fs_gcing(fscrypt_pagecache_folio(folio)); > > inode = mapping->host; > - sbi = F2FS_I_SB(inode); > > if (inode->i_ino == F2FS_META_INO(sbi) || > inode->i_ino == F2FS_NODE_INO(sbi) || > @@ -437,6 +431,65 @@ static void f2fs_write_end_io(struct bio *bio) > } > } > > +static void f2fs_cache_read_end_io(struct bio *bio) > +{ > + struct f2fs_cached_block *entry = F2FS_BIO(bio)->entry; > + struct f2fs_sb_info *sbi = entry->cache->sbi; > + enum count_type io_type = IS_META_CACHE(entry->cache) ? > + F2FS_RD_META : F2FS_RD_NODE; > + struct f2fs_cached_block *next; > + > + iostat_update_and_unbind_ctx(bio); > + > + if (time_to_inject(sbi, FAULT_READ_IO)) > + bio->bi_status = BLK_STS_IOERR; > + > + while (entry) { > + next = entry->next_entry; > + entry->next_entry = NULL; > + > + if (bio->bi_status == BLK_STS_OK) > + f2fs_cache_set_uptodate(entry); > + > + dec_page_count(sbi, io_type); > + > + f2fs_unlock_cache(entry); > + entry = next; > + } > + bio_put(bio); > +} > + > +static void f2fs_cache_write_end_io(struct bio *bio) > +{ > + struct f2fs_cached_block *entry = F2FS_BIO(bio)->entry; > + struct f2fs_sb_info *sbi = entry->cache->sbi; > + struct f2fs_cached_block *next; > + > + iostat_update_and_unbind_ctx(bio); > + > + if (time_to_inject(sbi, FAULT_WRITE_IO)) > + bio->bi_status = BLK_STS_IOERR; > + > + if (bio->bi_status != BLK_STS_OK) > + f2fs_stop_checkpoint(sbi, true, > + STOP_CP_REASON_WRITE_FAIL); > + > + while (entry) { > + next = entry->next_entry; > + entry->next_entry = NULL; > + > + dec_page_count(sbi, F2FS_WB_CP_DATA); > + > + if (!get_pages(sbi, F2FS_WB_CP_DATA) && > + wq_has_sleeper(&sbi->cp_wait)) > + wake_up(&sbi->cp_wait); > + > + f2fs_end_cache_writeback(entry); > + entry = next; > + } > + bio_put(bio); > +} > + > #ifdef CONFIG_BLK_DEV_ZONED > static void f2fs_zone_write_end_io(struct bio *bio) > { > @@ -444,7 +497,10 @@ static void f2fs_zone_write_end_io(struct bio *bio) > > bio->bi_private = io->bi_private; > complete(&io->zone_wait); > - f2fs_write_end_io(bio); > + if (io->fio.is_cache) > + f2fs_cache_write_end_io(bio); > + else > + f2fs_write_end_io(bio); > } > #endif > > @@ -531,12 +587,21 @@ static struct bio *__bio_alloc(struct f2fs_io_info *fio, int npages) > fio->op | fio->op_flags | f2fs_io_flags(fio), > GFP_NOIO, &f2fs_bioset); > bio->bi_iter.bi_sector = sector; > + F2FS_BIO(bio)->entry = NULL; > + bio->bi_private = NULL; > if (is_read_io(fio->op)) { > - bio->bi_end_io = f2fs_read_end_io; > - bio->bi_private = NULL; > + if (fio->is_cache) > + bio->bi_end_io = f2fs_cache_read_end_io; > + else > + bio->bi_end_io = f2fs_read_end_io; > } else { > - bio->bi_end_io = f2fs_write_end_io; > - bio->bi_private = sbi; > + if (fio->is_cache) { > + bio->bi_end_io = f2fs_cache_write_end_io; > + } else { > + bio->bi_end_io = f2fs_write_end_io; > + bio->bi_private = sbi; > + } > + > bio->bi_write_hint = f2fs_io_type_to_rw_hint(sbi, > fio->type, fio->temp); > bio->bi_write_stream = f2fs_io_type_to_write_stream(bdev, fio->type, > @@ -559,7 +624,7 @@ static void f2fs_set_bio_crypt_ctx(struct bio *bio, const struct inode *inode, > * The f2fs garbage collector sets ->encrypted_page when it wants to > * read/write raw data without encryption. > */ > - if (!fio || !fio->encrypted_page) > + if (!fio || (!fio->encrypted_page && !fio->is_cache)) > fscrypt_set_bio_crypt_ctx(bio, inode, > (loff_t)first_idx << inode->i_blkbits, > gfp_mask); > @@ -769,6 +834,53 @@ void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, > __submit_merged_write_cond(sbi, NULL, folio, 0, type, true); > } > > +static bool __has_merged_cache(struct bio *bio, > + struct f2fs_cached_block *target) > +{ > + struct f2fs_cached_block *entry; > + > + if (!bio) > + return false; > + > + entry = F2FS_BIO(bio)->entry; > + > + while (entry) { > + if (target && entry == target) > + return true; > + entry = entry->next_entry; > + } > + return false; > +} > + > +bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, > + enum page_type type) > +{ > + struct f2fs_sb_info *sbi = entry->cache->sbi; > + enum temp_type temp; > + bool ret = false; > + > + for (temp = HOT; temp < NR_TEMP_TYPE; temp++) { > + enum page_type btype = PAGE_TYPE_OF_BIO(type); > + struct f2fs_bio_info *io = sbi->write_io[btype] + temp; > + struct f2fs_lock_context lc; > + bool merged; > + > + f2fs_down_read_trace(&io->io_rwsem, &lc); > + merged = __has_merged_cache(io->bio, entry); > + f2fs_up_read_trace(&io->io_rwsem, &lc); > + > + if (merged) { > + __f2fs_submit_merged_write(sbi, type, temp); > + ret = true; > + } > + > + /* TODO: use HOT temp only for meta pages now. */ > + if (type >= META) > + break; > + } > + return ret; > +} > + > void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi) > { > f2fs_submit_merged_write(sbi, DATA); > @@ -832,6 +944,8 @@ static bool io_type_is_mergeable(struct f2fs_bio_info *io, > > if (io->fio.op != fio->op) > return false; > + if (io->fio.is_cache != fio->is_cache) > + return false; > return (io->fio.op_flags & mask) == (fio->op_flags & mask); > } > > @@ -1063,6 +1177,34 @@ static bool is_end_zone_blkaddr(struct f2fs_sb_info *sbi, block_t blkaddr) > f2fs_blkz_is_seq(sbi, devi, blkaddr) && > (blkaddr % sbi->blocks_per_blkz == sbi->blocks_per_blkz - 1); > } > + > +static void f2fs_wait_zone_io_completion(struct f2fs_sb_info *sbi, > + struct f2fs_bio_info *io, enum page_type btype) > +{ > + if (f2fs_sb_has_blkzoned(sbi) && btype < META && io->zone_pending_bio) { > + wait_for_completion_io(&io->zone_wait); > + bio_put(io->zone_pending_bio); > + io->zone_pending_bio = NULL; > + io->bi_private = NULL; > + } > +} > + > +static void f2fs_submit_zone_io(struct f2fs_sb_info *sbi, > + struct f2fs_io_info *fio, struct f2fs_bio_info *io, > + enum page_type btype) > +{ > + if (f2fs_sb_has_blkzoned(sbi) && btype < META && > + is_end_zone_blkaddr(sbi, fio->new_blkaddr)) { > + bio_get(io->bio); > + reinit_completion(&io->zone_wait); > + io->bi_private = io->bio->bi_private; > + io->bio->bi_private = io; > + io->bio->bi_end_io = f2fs_zone_write_end_io; > + io->zone_pending_bio = io->bio; > + __submit_merged_bio(io); > + } > + > +} > #endif > > void f2fs_submit_page_write(struct f2fs_io_info *fio) > @@ -1079,14 +1221,8 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio) > f2fs_down_write_trace(&io->io_rwsem, &lc); > next: > #ifdef CONFIG_BLK_DEV_ZONED > - if (f2fs_sb_has_blkzoned(sbi) && btype < META && io->zone_pending_bio) { > - wait_for_completion_io(&io->zone_wait); > - bio_put(io->zone_pending_bio); > - io->zone_pending_bio = NULL; > - io->bi_private = NULL; > - } > + f2fs_wait_zone_io_completion(sbi, io, btype); > #endif > - > if (fio->in_list) { > spin_lock(&io->io_lock); > if (list_empty(&io->io_list)) { > @@ -1141,16 +1277,109 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio) > > trace_f2fs_submit_folio_write(fio->folio, fio); > #ifdef CONFIG_BLK_DEV_ZONED > - if (f2fs_sb_has_blkzoned(sbi) && btype < META && > - is_end_zone_blkaddr(sbi, fio->new_blkaddr)) { > - bio_get(io->bio); > - reinit_completion(&io->zone_wait); > - io->bi_private = io->bio->bi_private; > - io->bio->bi_private = io; > - io->bio->bi_end_io = f2fs_zone_write_end_io; > - io->zone_pending_bio = io->bio; > + f2fs_submit_zone_io(sbi, fio, io, btype); > +#endif > + > + if (fio->in_list) > + goto next; > +out: > + if (is_sbi_flag_set(sbi, SBI_IS_SHUTDOWN) || > + !f2fs_is_checkpoint_ready(sbi)) > __submit_merged_bio(io); > + f2fs_up_write_trace(&io->io_rwsem, &lc); > +} > + > +static void f2fs_bio_add_cache(struct f2fs_io_info *fio, struct bio *bio) > +{ > + struct f2fs_bio *fbio = F2FS_BIO(bio); > + struct f2fs_cached_block *head = fbio->entry; > + struct f2fs_cached_block *new = fio->cache_entry; > + > + new->next_entry = head; > + fbio->entry = new; > +} > + > +int f2fs_submit_cache_read(struct f2fs_io_info *fio) > +{ > + struct f2fs_sb_info *sbi = fio->sbi; > + struct f2fs_cached_block *entry = fio->cache_entry; > + struct bio *bio; > + enum count_type io_type = IS_META_CACHE(entry->cache) ? > + F2FS_RD_META : F2FS_RD_NODE; > + > + if (!f2fs_is_valid_blkaddr(fio->sbi, fio->new_blkaddr, > + fio->is_por ? META_POR : (__is_meta_io(fio) ? > + META_GENERIC : DATA_GENERIC_ENHANCE))) > + return -EFSCORRUPTED; > + > + bio = __bio_alloc(fio, 1); > + > + bio_add_virt_nofail(bio, cache_address(entry), sbi->blocksize); > + f2fs_bio_add_cache(fio, bio); > + inc_page_count(sbi, io_type); > + > + f2fs_submit_read_bio(sbi, bio, fio->type); > + return 0; > +} > + > +void f2fs_submit_cache_write(struct f2fs_io_info *fio) > +{ > + struct f2fs_sb_info *sbi = fio->sbi; > + enum page_type btype = PAGE_TYPE_OF_BIO(fio->type); > + struct f2fs_bio_info *io = sbi->write_io[btype] + fio->temp; > + struct f2fs_lock_context lc; > + struct folio *folio; > + > + f2fs_bug_on(sbi, is_read_io(fio->op)); > + > + f2fs_down_write_trace(&io->io_rwsem, &lc); > +next: > +#ifdef CONFIG_BLK_DEV_ZONED > + f2fs_wait_zone_io_completion(sbi, io, btype); > +#endif > + if (fio->in_list) { > + spin_lock(&io->io_lock); > + if (list_empty(&io->io_list)) { > + spin_unlock(&io->io_lock); > + goto out; > + } > + fio = list_first_entry(&io->io_list, > + struct f2fs_io_info, list); > + list_del(&fio->list); > + spin_unlock(&io->io_lock); > } > + > + verify_fio_blkaddr(fio); > + > + fio->submitted = 1; > + inc_page_count(sbi, F2FS_WB_CP_DATA); > + > + if (io->bio && > + (!io_is_mergeable(sbi, io->bio, io, fio, io->last_block_in_bio, > + fio->new_blkaddr))) > + __submit_merged_bio(io); > +alloc_new: > + if (io->bio == NULL) { > + io->bio = __bio_alloc(fio, BIO_MAX_VECS); > + io->fio = *fio; > + } > + > + folio = cache_folio(fio->cache_entry); > + > + if (!bio_add_folio(io->bio, folio, sbi->blocksize, > + offset_in_folio(folio, cache_address(fio->cache_entry)))) { > + f2fs_bug_on(sbi, !F2FS_BIO(io->bio)->entry); > + > + __submit_merged_bio(io); > + goto alloc_new; > + } > + > + f2fs_bio_add_cache(fio, io->bio); > + > + io->last_block_in_bio = fio->new_blkaddr; > + > +#ifdef CONFIG_BLK_DEV_ZONED > + f2fs_submit_zone_io(sbi, fio, io, btype); > #endif > if (fio->in_list) > goto next; > diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h > index 1b96d8718c5c..8413983ea9d5 100644 > --- a/fs/f2fs/f2fs.h > +++ b/fs/f2fs/f2fs.h > @@ -221,6 +221,8 @@ struct f2fs_rwsem { > #endif > }; > > +#include "cache.h" > + > struct f2fs_mount_info { > unsigned long long opt; > block_t root_reserved_blocks; /* root reserved blocks */ > @@ -1370,8 +1372,10 @@ struct f2fs_io_info { > unsigned int is_por:1; /* indicate IO is from recovery or not */ > unsigned int encrypted:1; /* indicate file is encrypted */ > unsigned int meta_gc:1; /* require meta inode GC */ > + unsigned int is_cache:1; /* indicate IO is from internal cache */ > enum iostat_type io_type; /* io type */ > struct writeback_control *io_wbc; /* writeback control */ > + struct f2fs_cached_block *cache_entry; > struct bio **bio; /* bio for ipu */ > sector_t *last_block; /* last block number in bio */ > }; > @@ -1781,6 +1785,12 @@ struct f2fs_gc_kthread { > unsigned int boost_gc_greedy; > }; > > +struct f2fs_bio { > + struct work_struct work; > + struct f2fs_cached_block *entry; > + struct bio bio; > +}; > + > struct f2fs_sb_info { > struct super_block *sb; /* pointer to VFS super block */ > struct proc_dir_entry *s_proc; /* proc entry */ > @@ -2323,6 +2333,16 @@ static inline bool is_node_folio(struct folio *folio) > return folio->mapping == NODE_MAPPING(F2FS_F_SB(folio)); > } > > +static inline struct f2fs_bio *F2FS_BIO(struct bio *bio) > +{ > + return container_of(bio, struct f2fs_bio, bio); > +} > + > +static inline bool f2fs_is_cache_bio(struct bio *bio) > +{ > + return F2FS_BIO(bio)->entry != NULL; > +} > + > static inline bool is_sbi_flag_set(struct f2fs_sb_info *sbi, unsigned int type) > { > return test_bit(type, &sbi->s_flag); > @@ -4237,6 +4257,8 @@ void f2fs_submit_merged_write_cond(struct f2fs_sb_info *sbi, > nid_t ino, enum page_type type); > void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, > struct folio *folio, enum page_type type); > +bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, > + enum page_type type); > void f2fs_submit_merged_ipu_write(struct f2fs_sb_info *sbi, > struct bio **bio, struct folio *folio); > void f2fs_submit_all_merged_ipu_writes(struct f2fs_sb_info *sbi); > @@ -4244,6 +4266,8 @@ void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi); > int f2fs_submit_page_bio(struct f2fs_io_info *fio); > int f2fs_merge_page_bio(struct f2fs_io_info *fio); > void f2fs_submit_page_write(struct f2fs_io_info *fio); > +int f2fs_submit_cache_read(struct f2fs_io_info *fio); > +void f2fs_submit_cache_write(struct f2fs_io_info *fio); > struct block_device *f2fs_target_device(struct f2fs_sb_info *sbi, > block_t blk_addr, sector_t *sector); > int f2fs_target_device_index(struct f2fs_sb_info *sbi, block_t blkaddr); > diff --git a/fs/f2fs/iostat.h b/fs/f2fs/iostat.h > index 2025225b5bed..61c6bc8e3119 100644 > --- a/fs/f2fs/iostat.h > +++ b/fs/f2fs/iostat.h > @@ -60,6 +60,13 @@ static inline struct bio_post_read_ctx *get_post_read_ctx(struct bio *bio) > return iostat_ctx->post_read_ctx; > } > > +static inline void iostat_set_post_read_ctx(struct bio *bio, void *ctx) > +{ > + struct bio_iostat_ctx *iostat_ctx = bio->bi_private; > + > + iostat_ctx->post_read_ctx = ctx; > +} > + > extern void iostat_update_and_unbind_ctx(struct bio *bio); > extern void iostat_alloc_and_bind_ctx(struct f2fs_sb_info *sbi, > struct bio *bio, struct bio_post_read_ctx *ctx); > @@ -81,6 +88,10 @@ static inline struct bio_post_read_ctx *get_post_read_ctx(struct bio *bio) > { > return bio->bi_private; > } > +static inline void iostat_set_post_read_ctx(struct bio *bio, void *ctx) > +{ > + bio->bi_private = ctx; > +} > static inline int f2fs_init_iostat_processing(void) { return 0; } > static inline void f2fs_destroy_iostat_processing(void) {} > static inline int f2fs_init_iostat(struct f2fs_sb_info *sbi) { return 0; } > -- > 2.49.0 > > > > _______________________________________________ > Linux-f2fs-devel mailing list > Linux-f2fs-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [f2fs-dev] [PATCH v1 01/12] f2fs: cache: implement metadata cache 2026-08-20 5:08 ` Jaegeuk Kim via Linux-f2fs-devel @ 2026-08-20 5:13 ` Chao Yu via Linux-f2fs-devel -1 siblings, 0 replies; 34+ messages in thread From: Chao Yu @ 2026-08-20 5:13 UTC (permalink / raw) To: Jaegeuk Kim; +Cc: chao, linux-kernel, linux-f2fs-devel On 8/20/26 13:08, Jaegeuk Kim wrote: > On 08/20, Chao Yu via Linux-f2fs-devel wrote: >> This patch introduces the core metadata block caching infrastructure to >> manage f2fs metadata independently of the page cache. >> >> It implements: >> - core cache APIs: get, create, put, drop, backed by a radix tree and >> a single global LRU list. >> - support multiple status of cached block: LOCKED, UPTODATE, DIRTY, >> WRITEBACK, INLINE. >> - internal bio based read/write helpers with adjacent block vector merging. >> >> Signed-off-by: Chao Yu <chao@kernel.org> >> --- >> fs/f2fs/Makefile | 2 +- >> fs/f2fs/cache.c | 531 +++++++++++++++++++++++++++++++++++++++++++++++ >> fs/f2fs/cache.h | 179 ++++++++++++++++ >> fs/f2fs/data.c | 283 ++++++++++++++++++++++--- >> fs/f2fs/f2fs.h | 24 +++ >> fs/f2fs/iostat.h | 11 + >> 6 files changed, 1002 insertions(+), 28 deletions(-) >> create mode 100644 fs/f2fs/cache.c >> create mode 100644 fs/f2fs/cache.h >> >> diff --git a/fs/f2fs/Makefile b/fs/f2fs/Makefile >> index 8a7322d229e4..fbf49c30b066 100644 >> --- a/fs/f2fs/Makefile >> +++ b/fs/f2fs/Makefile >> @@ -3,7 +3,7 @@ obj-$(CONFIG_F2FS_FS) += f2fs.o >> >> f2fs-y := dir.o file.o inode.o namei.o hash.o super.o inline.o >> f2fs-y += checkpoint.o gc.o data.o node.o segment.o recovery.o >> -f2fs-y += shrinker.o extent_cache.o sysfs.o >> +f2fs-y += shrinker.o extent_cache.o sysfs.o cache.o >> f2fs-$(CONFIG_F2FS_STAT_FS) += debug.o >> f2fs-$(CONFIG_F2FS_FS_XATTR) += xattr.o >> f2fs-$(CONFIG_F2FS_FS_POSIX_ACL) += acl.o >> diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c >> new file mode 100644 >> index 000000000000..08bc658166f7 >> --- /dev/null >> +++ b/fs/f2fs/cache.c >> @@ -0,0 +1,531 @@ >> +// SPDX-License-Identifier: GPL-2.0 >> +/* >> + * Copyright (c) 2026 Google LLC >> + * Author: Chao Yu <chaseyu@google.com> >> + */ >> +#include "linux/spinlock.h" > > #include <linux/spinlock.h>? > May need to Move below? Oh, we don't need this line, let's remove. And will fix below all according to your suggestion. Thanks, > >> +#include <linux/fs.h> >> +#include <linux/f2fs_fs.h> >> +#include <linux/radix-tree.h> >> +#include <linux/slab.h> >> +#include <linux/list.h> >> +#include <linux/pagemap.h> >> +#include <linux/kthread.h> >> +#include <linux/freezer.h> >> +#include <linux/delay.h> >> +#include "f2fs.h" >> +#include "cache.h" >> +#include "node.h" >> +#include "segment.h" >> + >> +void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, >> + enum page_type type) >> +{ >> + /* in case the entry was truncated or on-going shrink */ >> + if (!entry->cache) >> + return; >> + >> + if (!f2fs_cache_test_writeback(entry)) >> + return; >> + >> + /* submit cached bio */ >> + f2fs_submit_merged_write_cache(entry, type); >> + >> + wait_on_bit_io(&entry->state, F2FS_BLOCK_WRITEBACK, >> + TASK_UNINTERRUPTIBLE); >> +} >> + >> +void f2fs_cache_wait_writeback(struct f2fs_cached_block *entry) >> +{ >> + /* in case the entry was truncated or on-going shrink */ >> + if (!entry->cache) >> + return; >> + >> + f2fs_cache_wait_writeback_cond(entry, >> + IS_META_CACHE(entry->cache) ? META : NODE); >> +} >> + >> +static void f2fs_cache_update_tag(struct f2fs_cached_block *entry, unsigned int src, >> + unsigned int dst) > > f2fs_cache_update_tag(entry, clear_from, set_to); > >> +{ >> + struct f2fs_cached_block_list *cache = entry->cache; >> + unsigned long flags; >> + >> + spin_lock_irqsave(&cache->tree_lock, flags); >> + if (src) > > if (clear_fom != F2FS_CACHE_TAG_NONE) > >> + radix_tree_tag_clear(&cache->root, entry->index, src); >> + if (dst) > > if (set_to != F2FS_CACHE_TAG_NONE) > >> + radix_tree_tag_set(&cache->root, entry->index, dst); >> + spin_unlock_irqrestore(&cache->tree_lock, flags); >> +} >> + >> +bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry) >> +{ >> + struct f2fs_cached_block_list *cache = entry->cache; >> + >> + if (!f2fs_cache_test_uptodate(entry)) >> + f2fs_cache_set_uptodate(entry); > > f2fs_cache_set_uptodate(entry); > >> + >> +#ifdef CONFIG_F2FS_CHECK_FS >> + if (cache->type == F2FS_NODE_CACHE && IS_INODE(cache_folio(entry))) >> + f2fs_inode_chksum_set(cache->sbi, cache_folio(entry)); >> +#endif >> + >> + if (f2fs_cache_test_dirty(entry)) >> + return false; >> + >> + if (!f2fs_cache_test_and_set_dirty(entry)) { >> + enum count_type type = IS_META_CACHE(cache) ? >> + F2FS_DIRTY_META : F2FS_DIRTY_NODES; >> + >> + f2fs_cache_update_tag(entry, 0, F2FS_CACHE_TAG_DIRTY); > > f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_NONE, F2FS_CACHE_TAG_DIRTY); > >> + inc_page_count(cache->sbi, type); >> + return true; >> + } >> + >> + return false; >> +} >> + >> +bool f2fs_clear_cache_dirty(struct f2fs_cached_block *entry) >> +{ >> + if (!f2fs_cache_test_dirty(entry)) >> + return false; >> + >> + f2fs_cache_clear_dirty(entry); >> + return true; >> +} >> + >> +static void __f2fs_drop_cache_dirty(struct f2fs_cached_block *entry, bool force) > > > static void __drop_cache_dirty(struct f2fs_cached_block *entry, bool force) > >> +{ >> + >> + struct f2fs_cached_block_list *cache = entry->cache; >> + enum count_type type = IS_META_CACHE(cache) ? >> + F2FS_DIRTY_META : F2FS_DIRTY_NODES; >> + >> + f2fs_cache_clear_uptodate(entry); >> + >> + if (!force && !f2fs_clear_cache_dirty(entry)) >> + return; >> + >> + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_DIRTY, 0); >> + dec_page_count(cache->sbi, type); >> +} >> + >> +void f2fs_drop_cache_dirty(struct f2fs_cached_block *entry) >> +{ >> + __f2fs_drop_cache_dirty(entry, false); >> +} >> + >> +void f2fs_force_clear_cache_dirty(struct f2fs_cached_block *entry) >> +{ >> + __f2fs_drop_cache_dirty(entry, true); >> +} >> + >> +void f2fs_start_cache_writeback(struct f2fs_cached_block *entry) >> +{ >> + f2fs_cache_set_writeback(entry); >> + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_DIRTY, >> + F2FS_CACHE_TAG_WRITEBACK); >> +} >> + >> +void f2fs_end_cache_writeback(struct f2fs_cached_block *entry) >> +{ >> + /* >> + * should call f2fs_cache_update_tag() before clearing writeback bit, >> + * in case f2fs_truncate_cache() set entry->cache to NULL. >> + */ >> + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_WRITEBACK, 0); >> + clear_and_wake_up_bit(F2FS_BLOCK_WRITEBACK, &entry->state); >> +} >> + >> +static int f2fs_cache_refcount(struct f2fs_cached_block *entry) >> +{ >> + return atomic_read(&entry->refcount); >> +} >> + >> +static void __f2fs_free_cache(struct f2fs_cached_block *entry) >> +{ >> + kfree(entry->data); >> + kfree(entry); >> +} >> + >> +static void f2fs_free_cache(struct f2fs_cached_block *entry) >> +{ >> + WARN_ON_ONCE(!list_empty(&entry->list)); >> + WARN_ON_ONCE(f2fs_cache_refcount(entry)); >> + __f2fs_free_cache(entry); >> +} >> + >> +void f2fs_cache_get(struct f2fs_cached_block *entry) >> +{ >> + atomic_inc(&entry->refcount); >> +} >> + >> +static bool f2fs_cache_put(struct f2fs_cached_block *entry) >> +{ >> + WARN_ON_ONCE(!f2fs_cache_refcount(entry)); >> + if (atomic_dec_and_test(&entry->refcount)) { >> + f2fs_free_cache(entry); >> + return true; >> + } >> + return false; >> +} >> + >> +static struct f2fs_cached_block *f2fs_create_cache( >> + struct f2fs_cached_block_list *cache, >> + unsigned long index, bool nofail) >> +{ >> + struct f2fs_cached_block *entry; >> + unsigned int flags = GFP_NOFS; >> + >> + if (nofail) >> + flags |= __GFP_NOFAIL; >> + >> + entry = kzalloc_obj(*entry, flags); >> + if (!entry) >> + return ERR_PTR(-ENOMEM); >> + >> + entry->data = kzalloc(cache->sbi->blocksize, flags); > > We don't need kzalloc() since we have an uptodate flag. > >> + if (!entry->data) { >> + kfree(entry); >> + return ERR_PTR(-ENOMEM); >> + } >> + >> + entry->index = index; >> + >> + atomic_set(&entry->refcount, 0); >> + entry->next_entry = NULL; >> + INIT_LIST_HEAD(&entry->list); >> + >> + entry->cache = cache; >> + >> + return entry; >> +} >> + >> +static struct f2fs_cached_block *f2fs_insert_cache( >> + struct f2fs_cached_block_list *cache, >> + unsigned long index, >> + struct f2fs_cached_block *new) >> +{ >> + struct f2fs_cached_block *e; >> + int ret; >> + unsigned long flags; >> + >> + ret = radix_tree_preload(GFP_NOFS | __GFP_NOFAIL); >> + f2fs_bug_on(cache->sbi, ret); >> + >> + spin_lock(&cache->list_lock); >> + spin_lock_irqsave(&cache->tree_lock, flags); >> + e = radix_tree_lookup(&cache->root, index); >> + if (!e) { >> + e = new; >> + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(e)); >> + >> + ret = radix_tree_insert(&cache->root, index, e); >> + f2fs_bug_on(cache->sbi, ret); >> + >> + /* radix tree referenced cache entry */ >> + f2fs_cache_get(e); >> + f2fs_bug_on(cache->sbi, !list_empty(&e->list)); >> + list_add_tail(&e->list, &cache->lru_list); >> + cache->num_entries++; >> + } >> + f2fs_cache_get(e); >> + spin_unlock_irqrestore(&cache->tree_lock, flags); >> + spin_unlock(&cache->list_lock); >> + radix_tree_preload_end(); >> + >> + if (new != e) { >> + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(new)); >> + __f2fs_free_cache(new); >> + } >> + >> + return e; >> +} >> + >> +struct f2fs_cached_block *f2fs_find_cache( >> + struct f2fs_cached_block_list *cache, >> + unsigned long index) >> +{ >> + struct f2fs_cached_block *entry; >> + unsigned long flags; >> + >> + spin_lock(&cache->list_lock); >> + spin_lock_irqsave(&cache->tree_lock, flags); >> + entry = radix_tree_lookup(&cache->root, index); >> + if (entry) { >> + f2fs_bug_on(cache->sbi, !f2fs_cache_refcount(entry)); >> + f2fs_cache_get(entry); >> + if (!list_empty(&entry->list)) >> + list_move_tail(&entry->list, &cache->lru_list); >> + >> + } else { >> + entry = ERR_PTR(-ENOENT); >> + } >> + spin_unlock_irqrestore(&cache->tree_lock, flags); >> + spin_unlock(&cache->list_lock); >> + >> + return entry; >> +} >> + >> +struct f2fs_cached_block *f2fs_grab_cache( >> + struct f2fs_cached_block_list *cache, >> + unsigned long index, int flags) >> + >> +{ >> + struct f2fs_cached_block *entry, *new; >> + bool create = flags & F2FS_CACHE_CREATE; >> + bool nofail = flags & F2FS_CACHE_NOFAIL; >> + bool lock = flags & F2FS_CACHE_LOCK; >> + >> +repeat: >> + entry = f2fs_find_cache(cache, index); >> + if (!IS_ERR(entry)) >> + goto found; >> + >> + if (!create) >> + return ERR_PTR(-ENOENT); >> + >> + new = f2fs_create_cache(cache, index, nofail); >> + if (IS_ERR(new)) >> + return new; >> + >> + entry = f2fs_insert_cache(cache, index, new); >> +found: >> + if (lock) { >> + f2fs_lock_cache(entry); >> + /* has been truncated */ >> + if (entry->cache != cache) { >> + f2fs_put_cache(entry, true); >> + goto repeat; >> + } >> + } >> + return entry; >> +} >> + >> +bool f2fs_trylock_cache(struct f2fs_cached_block *entry) >> +{ >> + return !test_and_set_bit(F2FS_BLOCK_LOCKED, &entry->state); >> +} >> + >> +void f2fs_lock_cache(struct f2fs_cached_block *entry) >> +{ >> + wait_on_bit_lock(&entry->state, F2FS_BLOCK_LOCKED, >> + TASK_UNINTERRUPTIBLE); >> +} >> + >> +void f2fs_unlock_cache(struct f2fs_cached_block *entry) >> +{ >> + clear_and_wake_up_bit(F2FS_BLOCK_LOCKED, &entry->state); >> +} >> + >> +bool f2fs_put_cache(struct f2fs_cached_block *entry, bool unlock) >> +{ >> + if (IS_ERR_OR_NULL(entry)) >> + return false; >> + if (unlock) >> + f2fs_unlock_cache(entry); >> + return f2fs_cache_put(entry); >> +} >> + >> +unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache, >> + struct f2fs_cached_block **entries, >> + pgoff_t *first_index, unsigned int max_nr, >> + int tag) >> +{ >> + unsigned long flags; >> + int nr, i; >> + >> + spin_lock_irqsave(&cache->tree_lock, flags); >> + nr = radix_tree_gang_lookup_tag(&cache->root, (void **)entries, >> + *first_index, max_nr, tag); >> + if (!nr) >> + goto out; >> + >> + for (i = 0; i < nr; i++) >> + f2fs_cache_get(entries[i]); >> + *first_index = entries[nr - 1]->index + 1; >> +out: >> + spin_unlock_irqrestore(&cache->tree_lock, flags); >> + return nr; >> +} >> + >> +void f2fs_cache_gang_release(struct f2fs_cached_block **entries, >> + unsigned int nr_entries) >> +{ >> + int i; >> + >> + for (i = 0; i < nr_entries; i++) >> + f2fs_put_cache(entries[i], false); >> +} >> + >> +void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache) >> +{ >> + unsigned long index = 0; >> + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; >> + int nr, i; >> + >> +next: >> + nr = f2fs_cache_gang_lookup_tag(cache, entries, &index, >> + F2FS_ONSTACK_CACHES, F2FS_CACHE_TAG_WRITEBACK); >> + if (!nr) >> + return; >> + >> + for (i = 0; i < nr; i++) >> + f2fs_cache_wait_writeback(entries[i]); >> + f2fs_cache_gang_release(entries, nr); >> + goto next; >> +} >> + >> +static void f2fs_do_truncate_cache(struct f2fs_cached_block *entry, >> + bool drop_dirty) >> +{ >> + struct f2fs_cached_block_list *cache = entry->cache; >> + unsigned long flags; >> + >> + if (!drop_dirty && >> + (f2fs_cache_test_dirty(entry) || >> + f2fs_cache_test_writeback(entry))) >> + return; >> + >> + f2fs_cache_wait_writeback(entry); >> + f2fs_drop_cache_dirty(entry); >> + >> + spin_lock(&cache->list_lock); >> + spin_lock_irqsave(&cache->tree_lock, flags); >> + >> + f2fs_bug_on(cache->sbi, !entry->cache); >> + if (!radix_tree_delete(&cache->root, entry->index)) >> + f2fs_bug_on(cache->sbi, !entry->cache); >> + >> + entry->cache = NULL; >> + cache->num_entries--; >> + >> + atomic_dec(&entry->refcount); >> + f2fs_bug_on(cache->sbi, !f2fs_cache_refcount(entry)); >> + >> + f2fs_bug_on(cache->sbi, list_empty(&entry->list)); >> + list_del_init(&entry->list); >> + >> + spin_unlock_irqrestore(&cache->tree_lock, flags); >> + spin_unlock(&cache->list_lock); >> +} >> + >> +static void f2fs_truncate_cache(struct f2fs_cached_block *entry, >> + bool drop_dirty) >> +{ >> + f2fs_lock_cache(entry); >> + if (entry->cache) >> + f2fs_do_truncate_cache(entry, drop_dirty); >> + f2fs_unlock_cache(entry); >> +} >> + >> +static void f2fs_drop_cache(struct f2fs_cached_block_list *cache, >> + block_t blkaddr, bool drop_dirty) >> +{ >> + struct f2fs_cached_block *entry; >> + >> + entry = f2fs_find_cache(cache, blkaddr); >> + if (IS_ERR(entry)) >> + return; >> + >> + f2fs_truncate_cache(entry, drop_dirty); >> + f2fs_put_cache(entry, false); >> +} >> + >> +void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache, >> + unsigned long start, unsigned long len, bool drop_dirty) >> +{ >> + unsigned long index = start; >> + unsigned long end = (ULONG_MAX - start < len) ? >> + ULONG_MAX : (start + len); >> + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; >> + unsigned long flags; >> + int nr, i; >> + >> + if (len == 1) >> + return f2fs_drop_cache(cache, index, drop_dirty); >> + >> +next: >> + spin_lock_irqsave(&cache->tree_lock, flags); >> + nr = radix_tree_gang_lookup(&cache->root, (void **)entries, index, >> + min((unsigned long)F2FS_ONSTACK_CACHES, end - index)); >> + if (!nr) >> + goto out_unlock; >> + >> + for (i = 0; i < nr; i++) { >> + struct f2fs_cached_block *entry = entries[i]; >> + >> + if (entry->index >= end) { >> + nr = i; >> + break; >> + } >> + f2fs_cache_get(entry); >> + } >> +out_unlock: >> + spin_unlock_irqrestore(&cache->tree_lock, flags); >> + if (!nr) >> + return; >> + >> + for (i = 0; i < nr; i++) { >> + struct f2fs_cached_block *entry = entries[i]; >> + >> + index = entry->index + 1; >> + >> + f2fs_truncate_cache(entry, drop_dirty); >> + } >> + f2fs_cache_gang_release(entries, nr); >> + >> + if (index < end) >> + goto next; >> +} >> + >> +int f2fs_init_cache(struct f2fs_sb_info *sbi, >> + struct f2fs_cached_block_list *cache, >> + enum f2fs_cache_type type) >> +{ >> + cache->sbi = sbi; >> + cache->type = type; >> + INIT_RADIX_TREE(&cache->root, GFP_ATOMIC); >> + spin_lock_init(&cache->tree_lock); >> + spin_lock_init(&cache->list_lock); >> + INIT_LIST_HEAD(&cache->lru_list); >> + cache->num_entries = 0; >> + >> + return 0; >> +} >> + >> +void f2fs_destroy_cache(struct f2fs_cached_block_list *cache) >> +{ >> + struct list_head *head = &cache->lru_list; >> + struct f2fs_cached_block *entry; >> + unsigned long flags; >> + >> + f2fs_cache_wait_on_all_writeback(cache); >> +next: >> + spin_lock(&cache->list_lock); >> + if (list_empty(head)) { >> + spin_unlock(&cache->list_lock); >> + return; >> + } >> + entry = list_first_entry(head, struct f2fs_cached_block, list); >> + >> + spin_lock_irqsave(&cache->tree_lock, flags); >> + radix_tree_delete(&cache->root, entry->index); >> + cache->num_entries--; >> + list_del_init(&entry->list); >> + spin_unlock_irqrestore(&cache->tree_lock, flags); >> + >> + spin_unlock(&cache->list_lock); >> + >> + /* wait on read cache IO */ >> + f2fs_lock_cache(entry); >> + /* wait on write cache IO */ >> + f2fs_cache_wait_writeback(entry); >> + f2fs_bug_on(cache->sbi, f2fs_cache_test_dirty(entry)); >> + f2fs_bug_on(cache->sbi, f2fs_cache_test_writeback(entry)); >> + f2fs_bug_on(cache->sbi, !list_empty(&entry->list)); >> + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(entry) != 1); >> + f2fs_put_cache(entry, true); >> + goto next; >> +} >> diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h >> new file mode 100644 >> index 000000000000..686a974008ae >> --- /dev/null >> +++ b/fs/f2fs/cache.h >> @@ -0,0 +1,179 @@ >> +/* SPDX-License-Identifier: GPL-2.0 */ >> +/* >> + * Copyright (c) 2026 Google LLC >> + * Author: Chao Yu <chaseyu@google.com> >> + */ >> +#ifndef _LINUX_F2FS_CACHE_H >> +#define _LINUX_F2FS_CACHE_H >> + >> +#include <linux/pagemap.h> >> +#include <linux/mm.h> >> +#include <linux/list.h> >> +#include <linux/radix-tree.h> >> +#include <linux/spinlock.h> >> +#include <linux/wait.h> >> +#include <linux/types.h> >> + >> +struct f2fs_rwsem; >> +struct f2fs_io_info; >> +enum page_type; >> + >> +/* Represents a single cached block (meta, node or compress) */ >> +struct f2fs_cached_block { >> + struct list_head list; /* LRU list head */ >> + struct f2fs_cached_block_list *cache; /* parent cache list */ >> + union { >> + /* chain for merged BIO */ >> + struct f2fs_cached_block *next_entry; >> + nid_t ino; /* inode number for compress cache */ >> + }; >> + unsigned long index; /* key in radix tree, (meta/compress: pba, node: nid) */ >> + unsigned long state; /* cache entry state (e.g., Dirty, UpToDate) */ >> + void *data; /* blocksize-aligned memory (4KB or 16KB) */ >> + atomic_t refcount; /* reference count */ >> +}; >> + >> +struct f2fs_sb_info; >> + >> +enum f2fs_cache_type { >> + F2FS_META_CACHE, >> + F2FS_NODE_CACHE, >> +}; >> + >> +/* Main cache control structure (per sb_info) */ >> +struct f2fs_cached_block_list { >> + struct f2fs_sb_info *sbi; /* Pointer to f2fs_sb_info */ >> + struct radix_tree_root root; /* Radix tree for cache lookup */ >> + spinlock_t tree_lock; /* Lock for radix tree */ >> + struct list_head lru_list; /* Single global LRU list */ >> + spinlock_t list_lock; /* Lock for LRU list */ >> + enum f2fs_cache_type type; /* Cache type (Node or Meta) */ >> + unsigned long num_entries; /* Current number of entries */ >> +}; >> + >> +#define IS_META_CACHE(cache) (cache->type == F2FS_META_CACHE) >> + >> +/* Flags for f2fs_cached_block state */ >> +enum f2fs_cached_state { >> + F2FS_BLOCK_LOCKED, /* cache entry is locked */ >> + F2FS_BLOCK_UPTODATE, /* cache data is valid */ >> + F2FS_BLOCK_DIRTY, /* cache data is dirty, need to writeback the data */ >> + F2FS_BLOCK_WRITEBACK, /* cache data is writeback state */ >> + F2FS_BLOCK_INLINE_DATA, /* indicate inline data */ >> +}; >> + >> +enum { >> + __F2FS_CACHE_CREATE, /* create the cache if there is no cache entry */ >> + __F2FS_CACHE_LOCK, /* get and lock the cache entry */ >> + __F2FS_CACHE_NOFAIL, /* do not allow failure */ >> +}; >> + >> +enum f2fs_cache_request_flag { >> + F2FS_CACHE_CREATE = 1 << __F2FS_CACHE_CREATE, >> + F2FS_CACHE_LOCK = 1 << __F2FS_CACHE_LOCK, >> + F2FS_CACHE_NOFAIL = 1 << __F2FS_CACHE_NOFAIL, >> +}; >> + >> +#define F2FS_CACHE_LOCK_CREATE (F2FS_CACHE_LOCK | F2FS_CACHE_CREATE) >> + >> +#define F2FS_ONSTACK_CACHES (32) >> + >> +#define F2FS_CACHE_FLAG_TEST_FUNC(name, flagname) \ >> +static inline bool f2fs_cache_test_##name( \ >> + const struct f2fs_cached_block *entry) \ >> +{ \ >> + return test_bit(F2FS_BLOCK_##flagname, &entry->state); \ >> +} \ >> + >> +#define F2FS_CACHE_FLAG_SET_FUNC(name, flagname) \ >> +static inline void f2fs_cache_set_##name( \ >> + struct f2fs_cached_block *entry) \ >> +{ \ >> + set_bit(F2FS_BLOCK_##flagname, &entry->state); \ >> +} \ >> + >> +#define F2FS_CACHE_FLAG_CLEAR_FUNC(name, flagname) \ >> +static inline void f2fs_cache_clear_##name( \ >> + struct f2fs_cached_block *entry) \ >> +{ \ >> + clear_bit(F2FS_BLOCK_##flagname, &entry->state); \ >> +} \ >> + >> +#define F2FS_CACHE_FLAG_TEST_AND_SET_FUNC(name, flagname) \ >> +static inline bool f2fs_cache_test_and_set_##name( \ >> + struct f2fs_cached_block *entry) \ >> +{ \ >> + return test_and_set_bit(F2FS_BLOCK_##flagname, &entry->state); \ >> +} \ >> + >> +F2FS_CACHE_FLAG_TEST_FUNC(locked, LOCKED); >> +F2FS_CACHE_FLAG_SET_FUNC(locked, LOCKED); >> +F2FS_CACHE_FLAG_CLEAR_FUNC(locked, LOCKED); >> + >> +F2FS_CACHE_FLAG_TEST_FUNC(uptodate, UPTODATE); >> +F2FS_CACHE_FLAG_SET_FUNC(uptodate, UPTODATE); >> +F2FS_CACHE_FLAG_CLEAR_FUNC(uptodate, UPTODATE); >> + >> +F2FS_CACHE_FLAG_TEST_FUNC(dirty, DIRTY); >> +F2FS_CACHE_FLAG_SET_FUNC(dirty, DIRTY); >> +F2FS_CACHE_FLAG_CLEAR_FUNC(dirty, DIRTY); >> +F2FS_CACHE_FLAG_TEST_AND_SET_FUNC(dirty, DIRTY); >> + >> +F2FS_CACHE_FLAG_TEST_FUNC(writeback, WRITEBACK); >> +F2FS_CACHE_FLAG_SET_FUNC(writeback, WRITEBACK); >> +F2FS_CACHE_FLAG_CLEAR_FUNC(writeback, WRITEBACK); >> + >> +F2FS_CACHE_FLAG_TEST_FUNC(inline, INLINE_DATA); >> +F2FS_CACHE_FLAG_SET_FUNC(inline, INLINE_DATA); >> +F2FS_CACHE_FLAG_CLEAR_FUNC(inline, INLINE_DATA); >> + >> +static inline void *cache_address(const struct f2fs_cached_block *entry) >> +{ >> + return entry->data; >> +} >> + >> +#define CACHED_NODE(entry) ((struct f2fs_node *)(cache_address(entry))) >> + >> +static inline struct folio *cache_folio(const struct f2fs_cached_block *entry) >> +{ >> + return virt_to_folio(entry->data); >> +} >> + >> +int f2fs_init_cache(struct f2fs_sb_info *sbi, >> + struct f2fs_cached_block_list *cache, >> + enum f2fs_cache_type type); >> +void f2fs_destroy_cache(struct f2fs_cached_block_list *cache); >> +void f2fs_cache_get(struct f2fs_cached_block *entry); >> +struct f2fs_cached_block *f2fs_find_cache( >> + struct f2fs_cached_block_list *cache, >> + unsigned long index); >> +#define F2FS_CACHE_TAG_NONE 0 >> +#define F2FS_CACHE_TAG_DIRTY 1 >> +#define F2FS_CACHE_TAG_WRITEBACK 2 >> + >> +bool f2fs_trylock_cache(struct f2fs_cached_block *entry); >> +void f2fs_lock_cache(struct f2fs_cached_block *entry); >> +void f2fs_unlock_cache(struct f2fs_cached_block *entry); >> +bool f2fs_put_cache(struct f2fs_cached_block *entry, bool unlock); >> +bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry); >> +bool f2fs_clear_cache_dirty(struct f2fs_cached_block *entry); >> +void f2fs_drop_cache_dirty(struct f2fs_cached_block *entry); >> +void f2fs_force_clear_cache_dirty(struct f2fs_cached_block *entry); >> +void f2fs_start_cache_writeback(struct f2fs_cached_block *entry); >> +void f2fs_end_cache_writeback(struct f2fs_cached_block *entry); >> +unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache, >> + struct f2fs_cached_block **results, pgoff_t *first_index, >> + unsigned int max_items, int tag); >> +void f2fs_cache_gang_release(struct f2fs_cached_block **entries, >> + unsigned int nr_entries); >> +int f2fs_writeback_cache(struct f2fs_cached_block_list *cache, bool sync); >> +void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache); >> +void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, >> + enum page_type type); >> +void f2fs_cache_wait_writeback(struct f2fs_cached_block *entry); >> +struct f2fs_cached_block *f2fs_grab_cache(struct f2fs_cached_block_list *cache, >> + unsigned long index, int flags); >> +void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache, >> + unsigned long start, unsigned long len, bool drop_dirty); >> + >> +#endif /* _LINUX_F2FS_CACHE_H */ >> diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c >> index 6ae0eb37d20f..09474790035b 100644 >> --- a/fs/f2fs/data.c >> +++ b/fs/f2fs/data.c >> @@ -41,11 +41,6 @@ struct f2fs_folio_state { >> unsigned int read_pages_pending; >> }; >> >> -struct f2fs_bio { >> - struct work_struct work; >> - struct bio bio; >> -}; >> - >> #define F2FS_BIO_POOL_SIZE NR_CURSEG_TYPE >> >> int __init f2fs_init_bioset(void) >> @@ -69,7 +64,6 @@ bool f2fs_is_cp_guaranteed(const struct folio *folio) >> return folio_test_f2fs_gcing(fscrypt_pagecache_folio(folio)); >> >> inode = mapping->host; >> - sbi = F2FS_I_SB(inode); >> >> if (inode->i_ino == F2FS_META_INO(sbi) || >> inode->i_ino == F2FS_NODE_INO(sbi) || >> @@ -437,6 +431,65 @@ static void f2fs_write_end_io(struct bio *bio) >> } >> } >> >> +static void f2fs_cache_read_end_io(struct bio *bio) >> +{ >> + struct f2fs_cached_block *entry = F2FS_BIO(bio)->entry; >> + struct f2fs_sb_info *sbi = entry->cache->sbi; >> + enum count_type io_type = IS_META_CACHE(entry->cache) ? >> + F2FS_RD_META : F2FS_RD_NODE; >> + struct f2fs_cached_block *next; >> + >> + iostat_update_and_unbind_ctx(bio); >> + >> + if (time_to_inject(sbi, FAULT_READ_IO)) >> + bio->bi_status = BLK_STS_IOERR; >> + >> + while (entry) { >> + next = entry->next_entry; >> + entry->next_entry = NULL; >> + >> + if (bio->bi_status == BLK_STS_OK) >> + f2fs_cache_set_uptodate(entry); >> + >> + dec_page_count(sbi, io_type); >> + >> + f2fs_unlock_cache(entry); >> + entry = next; >> + } >> + bio_put(bio); >> +} >> + >> +static void f2fs_cache_write_end_io(struct bio *bio) >> +{ >> + struct f2fs_cached_block *entry = F2FS_BIO(bio)->entry; >> + struct f2fs_sb_info *sbi = entry->cache->sbi; >> + struct f2fs_cached_block *next; >> + >> + iostat_update_and_unbind_ctx(bio); >> + >> + if (time_to_inject(sbi, FAULT_WRITE_IO)) >> + bio->bi_status = BLK_STS_IOERR; >> + >> + if (bio->bi_status != BLK_STS_OK) >> + f2fs_stop_checkpoint(sbi, true, >> + STOP_CP_REASON_WRITE_FAIL); >> + >> + while (entry) { >> + next = entry->next_entry; >> + entry->next_entry = NULL; >> + >> + dec_page_count(sbi, F2FS_WB_CP_DATA); >> + >> + if (!get_pages(sbi, F2FS_WB_CP_DATA) && >> + wq_has_sleeper(&sbi->cp_wait)) >> + wake_up(&sbi->cp_wait); >> + >> + f2fs_end_cache_writeback(entry); >> + entry = next; >> + } >> + bio_put(bio); >> +} >> + >> #ifdef CONFIG_BLK_DEV_ZONED >> static void f2fs_zone_write_end_io(struct bio *bio) >> { >> @@ -444,7 +497,10 @@ static void f2fs_zone_write_end_io(struct bio *bio) >> >> bio->bi_private = io->bi_private; >> complete(&io->zone_wait); >> - f2fs_write_end_io(bio); >> + if (io->fio.is_cache) >> + f2fs_cache_write_end_io(bio); >> + else >> + f2fs_write_end_io(bio); >> } >> #endif >> >> @@ -531,12 +587,21 @@ static struct bio *__bio_alloc(struct f2fs_io_info *fio, int npages) >> fio->op | fio->op_flags | f2fs_io_flags(fio), >> GFP_NOIO, &f2fs_bioset); >> bio->bi_iter.bi_sector = sector; >> + F2FS_BIO(bio)->entry = NULL; >> + bio->bi_private = NULL; >> if (is_read_io(fio->op)) { >> - bio->bi_end_io = f2fs_read_end_io; >> - bio->bi_private = NULL; >> + if (fio->is_cache) >> + bio->bi_end_io = f2fs_cache_read_end_io; >> + else >> + bio->bi_end_io = f2fs_read_end_io; >> } else { >> - bio->bi_end_io = f2fs_write_end_io; >> - bio->bi_private = sbi; >> + if (fio->is_cache) { >> + bio->bi_end_io = f2fs_cache_write_end_io; >> + } else { >> + bio->bi_end_io = f2fs_write_end_io; >> + bio->bi_private = sbi; >> + } >> + >> bio->bi_write_hint = f2fs_io_type_to_rw_hint(sbi, >> fio->type, fio->temp); >> bio->bi_write_stream = f2fs_io_type_to_write_stream(bdev, fio->type, >> @@ -559,7 +624,7 @@ static void f2fs_set_bio_crypt_ctx(struct bio *bio, const struct inode *inode, >> * The f2fs garbage collector sets ->encrypted_page when it wants to >> * read/write raw data without encryption. >> */ >> - if (!fio || !fio->encrypted_page) >> + if (!fio || (!fio->encrypted_page && !fio->is_cache)) >> fscrypt_set_bio_crypt_ctx(bio, inode, >> (loff_t)first_idx << inode->i_blkbits, >> gfp_mask); >> @@ -769,6 +834,53 @@ void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, >> __submit_merged_write_cond(sbi, NULL, folio, 0, type, true); >> } >> >> +static bool __has_merged_cache(struct bio *bio, >> + struct f2fs_cached_block *target) >> +{ >> + struct f2fs_cached_block *entry; >> + >> + if (!bio) >> + return false; >> + >> + entry = F2FS_BIO(bio)->entry; >> + >> + while (entry) { >> + if (target && entry == target) >> + return true; >> + entry = entry->next_entry; >> + } >> + return false; >> +} >> + >> +bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, >> + enum page_type type) >> +{ >> + struct f2fs_sb_info *sbi = entry->cache->sbi; >> + enum temp_type temp; >> + bool ret = false; >> + >> + for (temp = HOT; temp < NR_TEMP_TYPE; temp++) { >> + enum page_type btype = PAGE_TYPE_OF_BIO(type); >> + struct f2fs_bio_info *io = sbi->write_io[btype] + temp; >> + struct f2fs_lock_context lc; >> + bool merged; >> + >> + f2fs_down_read_trace(&io->io_rwsem, &lc); >> + merged = __has_merged_cache(io->bio, entry); >> + f2fs_up_read_trace(&io->io_rwsem, &lc); >> + >> + if (merged) { >> + __f2fs_submit_merged_write(sbi, type, temp); >> + ret = true; >> + } >> + >> + /* TODO: use HOT temp only for meta pages now. */ >> + if (type >= META) >> + break; >> + } >> + return ret; >> +} >> + >> void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi) >> { >> f2fs_submit_merged_write(sbi, DATA); >> @@ -832,6 +944,8 @@ static bool io_type_is_mergeable(struct f2fs_bio_info *io, >> >> if (io->fio.op != fio->op) >> return false; >> + if (io->fio.is_cache != fio->is_cache) >> + return false; >> return (io->fio.op_flags & mask) == (fio->op_flags & mask); >> } >> >> @@ -1063,6 +1177,34 @@ static bool is_end_zone_blkaddr(struct f2fs_sb_info *sbi, block_t blkaddr) >> f2fs_blkz_is_seq(sbi, devi, blkaddr) && >> (blkaddr % sbi->blocks_per_blkz == sbi->blocks_per_blkz - 1); >> } >> + >> +static void f2fs_wait_zone_io_completion(struct f2fs_sb_info *sbi, >> + struct f2fs_bio_info *io, enum page_type btype) >> +{ >> + if (f2fs_sb_has_blkzoned(sbi) && btype < META && io->zone_pending_bio) { >> + wait_for_completion_io(&io->zone_wait); >> + bio_put(io->zone_pending_bio); >> + io->zone_pending_bio = NULL; >> + io->bi_private = NULL; >> + } >> +} >> + >> +static void f2fs_submit_zone_io(struct f2fs_sb_info *sbi, >> + struct f2fs_io_info *fio, struct f2fs_bio_info *io, >> + enum page_type btype) >> +{ >> + if (f2fs_sb_has_blkzoned(sbi) && btype < META && >> + is_end_zone_blkaddr(sbi, fio->new_blkaddr)) { >> + bio_get(io->bio); >> + reinit_completion(&io->zone_wait); >> + io->bi_private = io->bio->bi_private; >> + io->bio->bi_private = io; >> + io->bio->bi_end_io = f2fs_zone_write_end_io; >> + io->zone_pending_bio = io->bio; >> + __submit_merged_bio(io); >> + } >> + >> +} >> #endif >> >> void f2fs_submit_page_write(struct f2fs_io_info *fio) >> @@ -1079,14 +1221,8 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio) >> f2fs_down_write_trace(&io->io_rwsem, &lc); >> next: >> #ifdef CONFIG_BLK_DEV_ZONED >> - if (f2fs_sb_has_blkzoned(sbi) && btype < META && io->zone_pending_bio) { >> - wait_for_completion_io(&io->zone_wait); >> - bio_put(io->zone_pending_bio); >> - io->zone_pending_bio = NULL; >> - io->bi_private = NULL; >> - } >> + f2fs_wait_zone_io_completion(sbi, io, btype); >> #endif >> - >> if (fio->in_list) { >> spin_lock(&io->io_lock); >> if (list_empty(&io->io_list)) { >> @@ -1141,16 +1277,109 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio) >> >> trace_f2fs_submit_folio_write(fio->folio, fio); >> #ifdef CONFIG_BLK_DEV_ZONED >> - if (f2fs_sb_has_blkzoned(sbi) && btype < META && >> - is_end_zone_blkaddr(sbi, fio->new_blkaddr)) { >> - bio_get(io->bio); >> - reinit_completion(&io->zone_wait); >> - io->bi_private = io->bio->bi_private; >> - io->bio->bi_private = io; >> - io->bio->bi_end_io = f2fs_zone_write_end_io; >> - io->zone_pending_bio = io->bio; >> + f2fs_submit_zone_io(sbi, fio, io, btype); >> +#endif >> + >> + if (fio->in_list) >> + goto next; >> +out: >> + if (is_sbi_flag_set(sbi, SBI_IS_SHUTDOWN) || >> + !f2fs_is_checkpoint_ready(sbi)) >> __submit_merged_bio(io); >> + f2fs_up_write_trace(&io->io_rwsem, &lc); >> +} >> + >> +static void f2fs_bio_add_cache(struct f2fs_io_info *fio, struct bio *bio) >> +{ >> + struct f2fs_bio *fbio = F2FS_BIO(bio); >> + struct f2fs_cached_block *head = fbio->entry; >> + struct f2fs_cached_block *new = fio->cache_entry; >> + >> + new->next_entry = head; >> + fbio->entry = new; >> +} >> + >> +int f2fs_submit_cache_read(struct f2fs_io_info *fio) >> +{ >> + struct f2fs_sb_info *sbi = fio->sbi; >> + struct f2fs_cached_block *entry = fio->cache_entry; >> + struct bio *bio; >> + enum count_type io_type = IS_META_CACHE(entry->cache) ? >> + F2FS_RD_META : F2FS_RD_NODE; >> + >> + if (!f2fs_is_valid_blkaddr(fio->sbi, fio->new_blkaddr, >> + fio->is_por ? META_POR : (__is_meta_io(fio) ? >> + META_GENERIC : DATA_GENERIC_ENHANCE))) >> + return -EFSCORRUPTED; >> + >> + bio = __bio_alloc(fio, 1); >> + >> + bio_add_virt_nofail(bio, cache_address(entry), sbi->blocksize); >> + f2fs_bio_add_cache(fio, bio); >> + inc_page_count(sbi, io_type); >> + >> + f2fs_submit_read_bio(sbi, bio, fio->type); >> + return 0; >> +} >> + >> +void f2fs_submit_cache_write(struct f2fs_io_info *fio) >> +{ >> + struct f2fs_sb_info *sbi = fio->sbi; >> + enum page_type btype = PAGE_TYPE_OF_BIO(fio->type); >> + struct f2fs_bio_info *io = sbi->write_io[btype] + fio->temp; >> + struct f2fs_lock_context lc; >> + struct folio *folio; >> + >> + f2fs_bug_on(sbi, is_read_io(fio->op)); >> + >> + f2fs_down_write_trace(&io->io_rwsem, &lc); >> +next: >> +#ifdef CONFIG_BLK_DEV_ZONED >> + f2fs_wait_zone_io_completion(sbi, io, btype); >> +#endif >> + if (fio->in_list) { >> + spin_lock(&io->io_lock); >> + if (list_empty(&io->io_list)) { >> + spin_unlock(&io->io_lock); >> + goto out; >> + } >> + fio = list_first_entry(&io->io_list, >> + struct f2fs_io_info, list); >> + list_del(&fio->list); >> + spin_unlock(&io->io_lock); >> } >> + >> + verify_fio_blkaddr(fio); >> + >> + fio->submitted = 1; >> + inc_page_count(sbi, F2FS_WB_CP_DATA); >> + >> + if (io->bio && >> + (!io_is_mergeable(sbi, io->bio, io, fio, io->last_block_in_bio, >> + fio->new_blkaddr))) >> + __submit_merged_bio(io); >> +alloc_new: >> + if (io->bio == NULL) { >> + io->bio = __bio_alloc(fio, BIO_MAX_VECS); >> + io->fio = *fio; >> + } >> + >> + folio = cache_folio(fio->cache_entry); >> + >> + if (!bio_add_folio(io->bio, folio, sbi->blocksize, >> + offset_in_folio(folio, cache_address(fio->cache_entry)))) { >> + f2fs_bug_on(sbi, !F2FS_BIO(io->bio)->entry); >> + >> + __submit_merged_bio(io); >> + goto alloc_new; >> + } >> + >> + f2fs_bio_add_cache(fio, io->bio); >> + >> + io->last_block_in_bio = fio->new_blkaddr; >> + >> +#ifdef CONFIG_BLK_DEV_ZONED >> + f2fs_submit_zone_io(sbi, fio, io, btype); >> #endif >> if (fio->in_list) >> goto next; >> diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h >> index 1b96d8718c5c..8413983ea9d5 100644 >> --- a/fs/f2fs/f2fs.h >> +++ b/fs/f2fs/f2fs.h >> @@ -221,6 +221,8 @@ struct f2fs_rwsem { >> #endif >> }; >> >> +#include "cache.h" >> + >> struct f2fs_mount_info { >> unsigned long long opt; >> block_t root_reserved_blocks; /* root reserved blocks */ >> @@ -1370,8 +1372,10 @@ struct f2fs_io_info { >> unsigned int is_por:1; /* indicate IO is from recovery or not */ >> unsigned int encrypted:1; /* indicate file is encrypted */ >> unsigned int meta_gc:1; /* require meta inode GC */ >> + unsigned int is_cache:1; /* indicate IO is from internal cache */ >> enum iostat_type io_type; /* io type */ >> struct writeback_control *io_wbc; /* writeback control */ >> + struct f2fs_cached_block *cache_entry; >> struct bio **bio; /* bio for ipu */ >> sector_t *last_block; /* last block number in bio */ >> }; >> @@ -1781,6 +1785,12 @@ struct f2fs_gc_kthread { >> unsigned int boost_gc_greedy; >> }; >> >> +struct f2fs_bio { >> + struct work_struct work; >> + struct f2fs_cached_block *entry; >> + struct bio bio; >> +}; >> + >> struct f2fs_sb_info { >> struct super_block *sb; /* pointer to VFS super block */ >> struct proc_dir_entry *s_proc; /* proc entry */ >> @@ -2323,6 +2333,16 @@ static inline bool is_node_folio(struct folio *folio) >> return folio->mapping == NODE_MAPPING(F2FS_F_SB(folio)); >> } >> >> +static inline struct f2fs_bio *F2FS_BIO(struct bio *bio) >> +{ >> + return container_of(bio, struct f2fs_bio, bio); >> +} >> + >> +static inline bool f2fs_is_cache_bio(struct bio *bio) >> +{ >> + return F2FS_BIO(bio)->entry != NULL; >> +} >> + >> static inline bool is_sbi_flag_set(struct f2fs_sb_info *sbi, unsigned int type) >> { >> return test_bit(type, &sbi->s_flag); >> @@ -4237,6 +4257,8 @@ void f2fs_submit_merged_write_cond(struct f2fs_sb_info *sbi, >> nid_t ino, enum page_type type); >> void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, >> struct folio *folio, enum page_type type); >> +bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, >> + enum page_type type); >> void f2fs_submit_merged_ipu_write(struct f2fs_sb_info *sbi, >> struct bio **bio, struct folio *folio); >> void f2fs_submit_all_merged_ipu_writes(struct f2fs_sb_info *sbi); >> @@ -4244,6 +4266,8 @@ void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi); >> int f2fs_submit_page_bio(struct f2fs_io_info *fio); >> int f2fs_merge_page_bio(struct f2fs_io_info *fio); >> void f2fs_submit_page_write(struct f2fs_io_info *fio); >> +int f2fs_submit_cache_read(struct f2fs_io_info *fio); >> +void f2fs_submit_cache_write(struct f2fs_io_info *fio); >> struct block_device *f2fs_target_device(struct f2fs_sb_info *sbi, >> block_t blk_addr, sector_t *sector); >> int f2fs_target_device_index(struct f2fs_sb_info *sbi, block_t blkaddr); >> diff --git a/fs/f2fs/iostat.h b/fs/f2fs/iostat.h >> index 2025225b5bed..61c6bc8e3119 100644 >> --- a/fs/f2fs/iostat.h >> +++ b/fs/f2fs/iostat.h >> @@ -60,6 +60,13 @@ static inline struct bio_post_read_ctx *get_post_read_ctx(struct bio *bio) >> return iostat_ctx->post_read_ctx; >> } >> >> +static inline void iostat_set_post_read_ctx(struct bio *bio, void *ctx) >> +{ >> + struct bio_iostat_ctx *iostat_ctx = bio->bi_private; >> + >> + iostat_ctx->post_read_ctx = ctx; >> +} >> + >> extern void iostat_update_and_unbind_ctx(struct bio *bio); >> extern void iostat_alloc_and_bind_ctx(struct f2fs_sb_info *sbi, >> struct bio *bio, struct bio_post_read_ctx *ctx); >> @@ -81,6 +88,10 @@ static inline struct bio_post_read_ctx *get_post_read_ctx(struct bio *bio) >> { >> return bio->bi_private; >> } >> +static inline void iostat_set_post_read_ctx(struct bio *bio, void *ctx) >> +{ >> + bio->bi_private = ctx; >> +} >> static inline int f2fs_init_iostat_processing(void) { return 0; } >> static inline void f2fs_destroy_iostat_processing(void) {} >> static inline int f2fs_init_iostat(struct f2fs_sb_info *sbi) { return 0; } >> -- >> 2.49.0 >> >> >> >> _______________________________________________ >> Linux-f2fs-devel mailing list >> Linux-f2fs-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [f2fs-dev] [PATCH v1 01/12] f2fs: cache: implement metadata cache @ 2026-08-20 5:13 ` Chao Yu via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Chao Yu via Linux-f2fs-devel @ 2026-08-20 5:13 UTC (permalink / raw) To: Jaegeuk Kim; +Cc: linux-kernel, linux-f2fs-devel On 8/20/26 13:08, Jaegeuk Kim wrote: > On 08/20, Chao Yu via Linux-f2fs-devel wrote: >> This patch introduces the core metadata block caching infrastructure to >> manage f2fs metadata independently of the page cache. >> >> It implements: >> - core cache APIs: get, create, put, drop, backed by a radix tree and >> a single global LRU list. >> - support multiple status of cached block: LOCKED, UPTODATE, DIRTY, >> WRITEBACK, INLINE. >> - internal bio based read/write helpers with adjacent block vector merging. >> >> Signed-off-by: Chao Yu <chao@kernel.org> >> --- >> fs/f2fs/Makefile | 2 +- >> fs/f2fs/cache.c | 531 +++++++++++++++++++++++++++++++++++++++++++++++ >> fs/f2fs/cache.h | 179 ++++++++++++++++ >> fs/f2fs/data.c | 283 ++++++++++++++++++++++--- >> fs/f2fs/f2fs.h | 24 +++ >> fs/f2fs/iostat.h | 11 + >> 6 files changed, 1002 insertions(+), 28 deletions(-) >> create mode 100644 fs/f2fs/cache.c >> create mode 100644 fs/f2fs/cache.h >> >> diff --git a/fs/f2fs/Makefile b/fs/f2fs/Makefile >> index 8a7322d229e4..fbf49c30b066 100644 >> --- a/fs/f2fs/Makefile >> +++ b/fs/f2fs/Makefile >> @@ -3,7 +3,7 @@ obj-$(CONFIG_F2FS_FS) += f2fs.o >> >> f2fs-y := dir.o file.o inode.o namei.o hash.o super.o inline.o >> f2fs-y += checkpoint.o gc.o data.o node.o segment.o recovery.o >> -f2fs-y += shrinker.o extent_cache.o sysfs.o >> +f2fs-y += shrinker.o extent_cache.o sysfs.o cache.o >> f2fs-$(CONFIG_F2FS_STAT_FS) += debug.o >> f2fs-$(CONFIG_F2FS_FS_XATTR) += xattr.o >> f2fs-$(CONFIG_F2FS_FS_POSIX_ACL) += acl.o >> diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c >> new file mode 100644 >> index 000000000000..08bc658166f7 >> --- /dev/null >> +++ b/fs/f2fs/cache.c >> @@ -0,0 +1,531 @@ >> +// SPDX-License-Identifier: GPL-2.0 >> +/* >> + * Copyright (c) 2026 Google LLC >> + * Author: Chao Yu <chaseyu@google.com> >> + */ >> +#include "linux/spinlock.h" > > #include <linux/spinlock.h>? > May need to Move below? Oh, we don't need this line, let's remove. And will fix below all according to your suggestion. Thanks, > >> +#include <linux/fs.h> >> +#include <linux/f2fs_fs.h> >> +#include <linux/radix-tree.h> >> +#include <linux/slab.h> >> +#include <linux/list.h> >> +#include <linux/pagemap.h> >> +#include <linux/kthread.h> >> +#include <linux/freezer.h> >> +#include <linux/delay.h> >> +#include "f2fs.h" >> +#include "cache.h" >> +#include "node.h" >> +#include "segment.h" >> + >> +void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, >> + enum page_type type) >> +{ >> + /* in case the entry was truncated or on-going shrink */ >> + if (!entry->cache) >> + return; >> + >> + if (!f2fs_cache_test_writeback(entry)) >> + return; >> + >> + /* submit cached bio */ >> + f2fs_submit_merged_write_cache(entry, type); >> + >> + wait_on_bit_io(&entry->state, F2FS_BLOCK_WRITEBACK, >> + TASK_UNINTERRUPTIBLE); >> +} >> + >> +void f2fs_cache_wait_writeback(struct f2fs_cached_block *entry) >> +{ >> + /* in case the entry was truncated or on-going shrink */ >> + if (!entry->cache) >> + return; >> + >> + f2fs_cache_wait_writeback_cond(entry, >> + IS_META_CACHE(entry->cache) ? META : NODE); >> +} >> + >> +static void f2fs_cache_update_tag(struct f2fs_cached_block *entry, unsigned int src, >> + unsigned int dst) > > f2fs_cache_update_tag(entry, clear_from, set_to); > >> +{ >> + struct f2fs_cached_block_list *cache = entry->cache; >> + unsigned long flags; >> + >> + spin_lock_irqsave(&cache->tree_lock, flags); >> + if (src) > > if (clear_fom != F2FS_CACHE_TAG_NONE) > >> + radix_tree_tag_clear(&cache->root, entry->index, src); >> + if (dst) > > if (set_to != F2FS_CACHE_TAG_NONE) > >> + radix_tree_tag_set(&cache->root, entry->index, dst); >> + spin_unlock_irqrestore(&cache->tree_lock, flags); >> +} >> + >> +bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry) >> +{ >> + struct f2fs_cached_block_list *cache = entry->cache; >> + >> + if (!f2fs_cache_test_uptodate(entry)) >> + f2fs_cache_set_uptodate(entry); > > f2fs_cache_set_uptodate(entry); > >> + >> +#ifdef CONFIG_F2FS_CHECK_FS >> + if (cache->type == F2FS_NODE_CACHE && IS_INODE(cache_folio(entry))) >> + f2fs_inode_chksum_set(cache->sbi, cache_folio(entry)); >> +#endif >> + >> + if (f2fs_cache_test_dirty(entry)) >> + return false; >> + >> + if (!f2fs_cache_test_and_set_dirty(entry)) { >> + enum count_type type = IS_META_CACHE(cache) ? >> + F2FS_DIRTY_META : F2FS_DIRTY_NODES; >> + >> + f2fs_cache_update_tag(entry, 0, F2FS_CACHE_TAG_DIRTY); > > f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_NONE, F2FS_CACHE_TAG_DIRTY); > >> + inc_page_count(cache->sbi, type); >> + return true; >> + } >> + >> + return false; >> +} >> + >> +bool f2fs_clear_cache_dirty(struct f2fs_cached_block *entry) >> +{ >> + if (!f2fs_cache_test_dirty(entry)) >> + return false; >> + >> + f2fs_cache_clear_dirty(entry); >> + return true; >> +} >> + >> +static void __f2fs_drop_cache_dirty(struct f2fs_cached_block *entry, bool force) > > > static void __drop_cache_dirty(struct f2fs_cached_block *entry, bool force) > >> +{ >> + >> + struct f2fs_cached_block_list *cache = entry->cache; >> + enum count_type type = IS_META_CACHE(cache) ? >> + F2FS_DIRTY_META : F2FS_DIRTY_NODES; >> + >> + f2fs_cache_clear_uptodate(entry); >> + >> + if (!force && !f2fs_clear_cache_dirty(entry)) >> + return; >> + >> + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_DIRTY, 0); >> + dec_page_count(cache->sbi, type); >> +} >> + >> +void f2fs_drop_cache_dirty(struct f2fs_cached_block *entry) >> +{ >> + __f2fs_drop_cache_dirty(entry, false); >> +} >> + >> +void f2fs_force_clear_cache_dirty(struct f2fs_cached_block *entry) >> +{ >> + __f2fs_drop_cache_dirty(entry, true); >> +} >> + >> +void f2fs_start_cache_writeback(struct f2fs_cached_block *entry) >> +{ >> + f2fs_cache_set_writeback(entry); >> + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_DIRTY, >> + F2FS_CACHE_TAG_WRITEBACK); >> +} >> + >> +void f2fs_end_cache_writeback(struct f2fs_cached_block *entry) >> +{ >> + /* >> + * should call f2fs_cache_update_tag() before clearing writeback bit, >> + * in case f2fs_truncate_cache() set entry->cache to NULL. >> + */ >> + f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_WRITEBACK, 0); >> + clear_and_wake_up_bit(F2FS_BLOCK_WRITEBACK, &entry->state); >> +} >> + >> +static int f2fs_cache_refcount(struct f2fs_cached_block *entry) >> +{ >> + return atomic_read(&entry->refcount); >> +} >> + >> +static void __f2fs_free_cache(struct f2fs_cached_block *entry) >> +{ >> + kfree(entry->data); >> + kfree(entry); >> +} >> + >> +static void f2fs_free_cache(struct f2fs_cached_block *entry) >> +{ >> + WARN_ON_ONCE(!list_empty(&entry->list)); >> + WARN_ON_ONCE(f2fs_cache_refcount(entry)); >> + __f2fs_free_cache(entry); >> +} >> + >> +void f2fs_cache_get(struct f2fs_cached_block *entry) >> +{ >> + atomic_inc(&entry->refcount); >> +} >> + >> +static bool f2fs_cache_put(struct f2fs_cached_block *entry) >> +{ >> + WARN_ON_ONCE(!f2fs_cache_refcount(entry)); >> + if (atomic_dec_and_test(&entry->refcount)) { >> + f2fs_free_cache(entry); >> + return true; >> + } >> + return false; >> +} >> + >> +static struct f2fs_cached_block *f2fs_create_cache( >> + struct f2fs_cached_block_list *cache, >> + unsigned long index, bool nofail) >> +{ >> + struct f2fs_cached_block *entry; >> + unsigned int flags = GFP_NOFS; >> + >> + if (nofail) >> + flags |= __GFP_NOFAIL; >> + >> + entry = kzalloc_obj(*entry, flags); >> + if (!entry) >> + return ERR_PTR(-ENOMEM); >> + >> + entry->data = kzalloc(cache->sbi->blocksize, flags); > > We don't need kzalloc() since we have an uptodate flag. > >> + if (!entry->data) { >> + kfree(entry); >> + return ERR_PTR(-ENOMEM); >> + } >> + >> + entry->index = index; >> + >> + atomic_set(&entry->refcount, 0); >> + entry->next_entry = NULL; >> + INIT_LIST_HEAD(&entry->list); >> + >> + entry->cache = cache; >> + >> + return entry; >> +} >> + >> +static struct f2fs_cached_block *f2fs_insert_cache( >> + struct f2fs_cached_block_list *cache, >> + unsigned long index, >> + struct f2fs_cached_block *new) >> +{ >> + struct f2fs_cached_block *e; >> + int ret; >> + unsigned long flags; >> + >> + ret = radix_tree_preload(GFP_NOFS | __GFP_NOFAIL); >> + f2fs_bug_on(cache->sbi, ret); >> + >> + spin_lock(&cache->list_lock); >> + spin_lock_irqsave(&cache->tree_lock, flags); >> + e = radix_tree_lookup(&cache->root, index); >> + if (!e) { >> + e = new; >> + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(e)); >> + >> + ret = radix_tree_insert(&cache->root, index, e); >> + f2fs_bug_on(cache->sbi, ret); >> + >> + /* radix tree referenced cache entry */ >> + f2fs_cache_get(e); >> + f2fs_bug_on(cache->sbi, !list_empty(&e->list)); >> + list_add_tail(&e->list, &cache->lru_list); >> + cache->num_entries++; >> + } >> + f2fs_cache_get(e); >> + spin_unlock_irqrestore(&cache->tree_lock, flags); >> + spin_unlock(&cache->list_lock); >> + radix_tree_preload_end(); >> + >> + if (new != e) { >> + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(new)); >> + __f2fs_free_cache(new); >> + } >> + >> + return e; >> +} >> + >> +struct f2fs_cached_block *f2fs_find_cache( >> + struct f2fs_cached_block_list *cache, >> + unsigned long index) >> +{ >> + struct f2fs_cached_block *entry; >> + unsigned long flags; >> + >> + spin_lock(&cache->list_lock); >> + spin_lock_irqsave(&cache->tree_lock, flags); >> + entry = radix_tree_lookup(&cache->root, index); >> + if (entry) { >> + f2fs_bug_on(cache->sbi, !f2fs_cache_refcount(entry)); >> + f2fs_cache_get(entry); >> + if (!list_empty(&entry->list)) >> + list_move_tail(&entry->list, &cache->lru_list); >> + >> + } else { >> + entry = ERR_PTR(-ENOENT); >> + } >> + spin_unlock_irqrestore(&cache->tree_lock, flags); >> + spin_unlock(&cache->list_lock); >> + >> + return entry; >> +} >> + >> +struct f2fs_cached_block *f2fs_grab_cache( >> + struct f2fs_cached_block_list *cache, >> + unsigned long index, int flags) >> + >> +{ >> + struct f2fs_cached_block *entry, *new; >> + bool create = flags & F2FS_CACHE_CREATE; >> + bool nofail = flags & F2FS_CACHE_NOFAIL; >> + bool lock = flags & F2FS_CACHE_LOCK; >> + >> +repeat: >> + entry = f2fs_find_cache(cache, index); >> + if (!IS_ERR(entry)) >> + goto found; >> + >> + if (!create) >> + return ERR_PTR(-ENOENT); >> + >> + new = f2fs_create_cache(cache, index, nofail); >> + if (IS_ERR(new)) >> + return new; >> + >> + entry = f2fs_insert_cache(cache, index, new); >> +found: >> + if (lock) { >> + f2fs_lock_cache(entry); >> + /* has been truncated */ >> + if (entry->cache != cache) { >> + f2fs_put_cache(entry, true); >> + goto repeat; >> + } >> + } >> + return entry; >> +} >> + >> +bool f2fs_trylock_cache(struct f2fs_cached_block *entry) >> +{ >> + return !test_and_set_bit(F2FS_BLOCK_LOCKED, &entry->state); >> +} >> + >> +void f2fs_lock_cache(struct f2fs_cached_block *entry) >> +{ >> + wait_on_bit_lock(&entry->state, F2FS_BLOCK_LOCKED, >> + TASK_UNINTERRUPTIBLE); >> +} >> + >> +void f2fs_unlock_cache(struct f2fs_cached_block *entry) >> +{ >> + clear_and_wake_up_bit(F2FS_BLOCK_LOCKED, &entry->state); >> +} >> + >> +bool f2fs_put_cache(struct f2fs_cached_block *entry, bool unlock) >> +{ >> + if (IS_ERR_OR_NULL(entry)) >> + return false; >> + if (unlock) >> + f2fs_unlock_cache(entry); >> + return f2fs_cache_put(entry); >> +} >> + >> +unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache, >> + struct f2fs_cached_block **entries, >> + pgoff_t *first_index, unsigned int max_nr, >> + int tag) >> +{ >> + unsigned long flags; >> + int nr, i; >> + >> + spin_lock_irqsave(&cache->tree_lock, flags); >> + nr = radix_tree_gang_lookup_tag(&cache->root, (void **)entries, >> + *first_index, max_nr, tag); >> + if (!nr) >> + goto out; >> + >> + for (i = 0; i < nr; i++) >> + f2fs_cache_get(entries[i]); >> + *first_index = entries[nr - 1]->index + 1; >> +out: >> + spin_unlock_irqrestore(&cache->tree_lock, flags); >> + return nr; >> +} >> + >> +void f2fs_cache_gang_release(struct f2fs_cached_block **entries, >> + unsigned int nr_entries) >> +{ >> + int i; >> + >> + for (i = 0; i < nr_entries; i++) >> + f2fs_put_cache(entries[i], false); >> +} >> + >> +void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache) >> +{ >> + unsigned long index = 0; >> + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; >> + int nr, i; >> + >> +next: >> + nr = f2fs_cache_gang_lookup_tag(cache, entries, &index, >> + F2FS_ONSTACK_CACHES, F2FS_CACHE_TAG_WRITEBACK); >> + if (!nr) >> + return; >> + >> + for (i = 0; i < nr; i++) >> + f2fs_cache_wait_writeback(entries[i]); >> + f2fs_cache_gang_release(entries, nr); >> + goto next; >> +} >> + >> +static void f2fs_do_truncate_cache(struct f2fs_cached_block *entry, >> + bool drop_dirty) >> +{ >> + struct f2fs_cached_block_list *cache = entry->cache; >> + unsigned long flags; >> + >> + if (!drop_dirty && >> + (f2fs_cache_test_dirty(entry) || >> + f2fs_cache_test_writeback(entry))) >> + return; >> + >> + f2fs_cache_wait_writeback(entry); >> + f2fs_drop_cache_dirty(entry); >> + >> + spin_lock(&cache->list_lock); >> + spin_lock_irqsave(&cache->tree_lock, flags); >> + >> + f2fs_bug_on(cache->sbi, !entry->cache); >> + if (!radix_tree_delete(&cache->root, entry->index)) >> + f2fs_bug_on(cache->sbi, !entry->cache); >> + >> + entry->cache = NULL; >> + cache->num_entries--; >> + >> + atomic_dec(&entry->refcount); >> + f2fs_bug_on(cache->sbi, !f2fs_cache_refcount(entry)); >> + >> + f2fs_bug_on(cache->sbi, list_empty(&entry->list)); >> + list_del_init(&entry->list); >> + >> + spin_unlock_irqrestore(&cache->tree_lock, flags); >> + spin_unlock(&cache->list_lock); >> +} >> + >> +static void f2fs_truncate_cache(struct f2fs_cached_block *entry, >> + bool drop_dirty) >> +{ >> + f2fs_lock_cache(entry); >> + if (entry->cache) >> + f2fs_do_truncate_cache(entry, drop_dirty); >> + f2fs_unlock_cache(entry); >> +} >> + >> +static void f2fs_drop_cache(struct f2fs_cached_block_list *cache, >> + block_t blkaddr, bool drop_dirty) >> +{ >> + struct f2fs_cached_block *entry; >> + >> + entry = f2fs_find_cache(cache, blkaddr); >> + if (IS_ERR(entry)) >> + return; >> + >> + f2fs_truncate_cache(entry, drop_dirty); >> + f2fs_put_cache(entry, false); >> +} >> + >> +void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache, >> + unsigned long start, unsigned long len, bool drop_dirty) >> +{ >> + unsigned long index = start; >> + unsigned long end = (ULONG_MAX - start < len) ? >> + ULONG_MAX : (start + len); >> + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; >> + unsigned long flags; >> + int nr, i; >> + >> + if (len == 1) >> + return f2fs_drop_cache(cache, index, drop_dirty); >> + >> +next: >> + spin_lock_irqsave(&cache->tree_lock, flags); >> + nr = radix_tree_gang_lookup(&cache->root, (void **)entries, index, >> + min((unsigned long)F2FS_ONSTACK_CACHES, end - index)); >> + if (!nr) >> + goto out_unlock; >> + >> + for (i = 0; i < nr; i++) { >> + struct f2fs_cached_block *entry = entries[i]; >> + >> + if (entry->index >= end) { >> + nr = i; >> + break; >> + } >> + f2fs_cache_get(entry); >> + } >> +out_unlock: >> + spin_unlock_irqrestore(&cache->tree_lock, flags); >> + if (!nr) >> + return; >> + >> + for (i = 0; i < nr; i++) { >> + struct f2fs_cached_block *entry = entries[i]; >> + >> + index = entry->index + 1; >> + >> + f2fs_truncate_cache(entry, drop_dirty); >> + } >> + f2fs_cache_gang_release(entries, nr); >> + >> + if (index < end) >> + goto next; >> +} >> + >> +int f2fs_init_cache(struct f2fs_sb_info *sbi, >> + struct f2fs_cached_block_list *cache, >> + enum f2fs_cache_type type) >> +{ >> + cache->sbi = sbi; >> + cache->type = type; >> + INIT_RADIX_TREE(&cache->root, GFP_ATOMIC); >> + spin_lock_init(&cache->tree_lock); >> + spin_lock_init(&cache->list_lock); >> + INIT_LIST_HEAD(&cache->lru_list); >> + cache->num_entries = 0; >> + >> + return 0; >> +} >> + >> +void f2fs_destroy_cache(struct f2fs_cached_block_list *cache) >> +{ >> + struct list_head *head = &cache->lru_list; >> + struct f2fs_cached_block *entry; >> + unsigned long flags; >> + >> + f2fs_cache_wait_on_all_writeback(cache); >> +next: >> + spin_lock(&cache->list_lock); >> + if (list_empty(head)) { >> + spin_unlock(&cache->list_lock); >> + return; >> + } >> + entry = list_first_entry(head, struct f2fs_cached_block, list); >> + >> + spin_lock_irqsave(&cache->tree_lock, flags); >> + radix_tree_delete(&cache->root, entry->index); >> + cache->num_entries--; >> + list_del_init(&entry->list); >> + spin_unlock_irqrestore(&cache->tree_lock, flags); >> + >> + spin_unlock(&cache->list_lock); >> + >> + /* wait on read cache IO */ >> + f2fs_lock_cache(entry); >> + /* wait on write cache IO */ >> + f2fs_cache_wait_writeback(entry); >> + f2fs_bug_on(cache->sbi, f2fs_cache_test_dirty(entry)); >> + f2fs_bug_on(cache->sbi, f2fs_cache_test_writeback(entry)); >> + f2fs_bug_on(cache->sbi, !list_empty(&entry->list)); >> + f2fs_bug_on(cache->sbi, f2fs_cache_refcount(entry) != 1); >> + f2fs_put_cache(entry, true); >> + goto next; >> +} >> diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h >> new file mode 100644 >> index 000000000000..686a974008ae >> --- /dev/null >> +++ b/fs/f2fs/cache.h >> @@ -0,0 +1,179 @@ >> +/* SPDX-License-Identifier: GPL-2.0 */ >> +/* >> + * Copyright (c) 2026 Google LLC >> + * Author: Chao Yu <chaseyu@google.com> >> + */ >> +#ifndef _LINUX_F2FS_CACHE_H >> +#define _LINUX_F2FS_CACHE_H >> + >> +#include <linux/pagemap.h> >> +#include <linux/mm.h> >> +#include <linux/list.h> >> +#include <linux/radix-tree.h> >> +#include <linux/spinlock.h> >> +#include <linux/wait.h> >> +#include <linux/types.h> >> + >> +struct f2fs_rwsem; >> +struct f2fs_io_info; >> +enum page_type; >> + >> +/* Represents a single cached block (meta, node or compress) */ >> +struct f2fs_cached_block { >> + struct list_head list; /* LRU list head */ >> + struct f2fs_cached_block_list *cache; /* parent cache list */ >> + union { >> + /* chain for merged BIO */ >> + struct f2fs_cached_block *next_entry; >> + nid_t ino; /* inode number for compress cache */ >> + }; >> + unsigned long index; /* key in radix tree, (meta/compress: pba, node: nid) */ >> + unsigned long state; /* cache entry state (e.g., Dirty, UpToDate) */ >> + void *data; /* blocksize-aligned memory (4KB or 16KB) */ >> + atomic_t refcount; /* reference count */ >> +}; >> + >> +struct f2fs_sb_info; >> + >> +enum f2fs_cache_type { >> + F2FS_META_CACHE, >> + F2FS_NODE_CACHE, >> +}; >> + >> +/* Main cache control structure (per sb_info) */ >> +struct f2fs_cached_block_list { >> + struct f2fs_sb_info *sbi; /* Pointer to f2fs_sb_info */ >> + struct radix_tree_root root; /* Radix tree for cache lookup */ >> + spinlock_t tree_lock; /* Lock for radix tree */ >> + struct list_head lru_list; /* Single global LRU list */ >> + spinlock_t list_lock; /* Lock for LRU list */ >> + enum f2fs_cache_type type; /* Cache type (Node or Meta) */ >> + unsigned long num_entries; /* Current number of entries */ >> +}; >> + >> +#define IS_META_CACHE(cache) (cache->type == F2FS_META_CACHE) >> + >> +/* Flags for f2fs_cached_block state */ >> +enum f2fs_cached_state { >> + F2FS_BLOCK_LOCKED, /* cache entry is locked */ >> + F2FS_BLOCK_UPTODATE, /* cache data is valid */ >> + F2FS_BLOCK_DIRTY, /* cache data is dirty, need to writeback the data */ >> + F2FS_BLOCK_WRITEBACK, /* cache data is writeback state */ >> + F2FS_BLOCK_INLINE_DATA, /* indicate inline data */ >> +}; >> + >> +enum { >> + __F2FS_CACHE_CREATE, /* create the cache if there is no cache entry */ >> + __F2FS_CACHE_LOCK, /* get and lock the cache entry */ >> + __F2FS_CACHE_NOFAIL, /* do not allow failure */ >> +}; >> + >> +enum f2fs_cache_request_flag { >> + F2FS_CACHE_CREATE = 1 << __F2FS_CACHE_CREATE, >> + F2FS_CACHE_LOCK = 1 << __F2FS_CACHE_LOCK, >> + F2FS_CACHE_NOFAIL = 1 << __F2FS_CACHE_NOFAIL, >> +}; >> + >> +#define F2FS_CACHE_LOCK_CREATE (F2FS_CACHE_LOCK | F2FS_CACHE_CREATE) >> + >> +#define F2FS_ONSTACK_CACHES (32) >> + >> +#define F2FS_CACHE_FLAG_TEST_FUNC(name, flagname) \ >> +static inline bool f2fs_cache_test_##name( \ >> + const struct f2fs_cached_block *entry) \ >> +{ \ >> + return test_bit(F2FS_BLOCK_##flagname, &entry->state); \ >> +} \ >> + >> +#define F2FS_CACHE_FLAG_SET_FUNC(name, flagname) \ >> +static inline void f2fs_cache_set_##name( \ >> + struct f2fs_cached_block *entry) \ >> +{ \ >> + set_bit(F2FS_BLOCK_##flagname, &entry->state); \ >> +} \ >> + >> +#define F2FS_CACHE_FLAG_CLEAR_FUNC(name, flagname) \ >> +static inline void f2fs_cache_clear_##name( \ >> + struct f2fs_cached_block *entry) \ >> +{ \ >> + clear_bit(F2FS_BLOCK_##flagname, &entry->state); \ >> +} \ >> + >> +#define F2FS_CACHE_FLAG_TEST_AND_SET_FUNC(name, flagname) \ >> +static inline bool f2fs_cache_test_and_set_##name( \ >> + struct f2fs_cached_block *entry) \ >> +{ \ >> + return test_and_set_bit(F2FS_BLOCK_##flagname, &entry->state); \ >> +} \ >> + >> +F2FS_CACHE_FLAG_TEST_FUNC(locked, LOCKED); >> +F2FS_CACHE_FLAG_SET_FUNC(locked, LOCKED); >> +F2FS_CACHE_FLAG_CLEAR_FUNC(locked, LOCKED); >> + >> +F2FS_CACHE_FLAG_TEST_FUNC(uptodate, UPTODATE); >> +F2FS_CACHE_FLAG_SET_FUNC(uptodate, UPTODATE); >> +F2FS_CACHE_FLAG_CLEAR_FUNC(uptodate, UPTODATE); >> + >> +F2FS_CACHE_FLAG_TEST_FUNC(dirty, DIRTY); >> +F2FS_CACHE_FLAG_SET_FUNC(dirty, DIRTY); >> +F2FS_CACHE_FLAG_CLEAR_FUNC(dirty, DIRTY); >> +F2FS_CACHE_FLAG_TEST_AND_SET_FUNC(dirty, DIRTY); >> + >> +F2FS_CACHE_FLAG_TEST_FUNC(writeback, WRITEBACK); >> +F2FS_CACHE_FLAG_SET_FUNC(writeback, WRITEBACK); >> +F2FS_CACHE_FLAG_CLEAR_FUNC(writeback, WRITEBACK); >> + >> +F2FS_CACHE_FLAG_TEST_FUNC(inline, INLINE_DATA); >> +F2FS_CACHE_FLAG_SET_FUNC(inline, INLINE_DATA); >> +F2FS_CACHE_FLAG_CLEAR_FUNC(inline, INLINE_DATA); >> + >> +static inline void *cache_address(const struct f2fs_cached_block *entry) >> +{ >> + return entry->data; >> +} >> + >> +#define CACHED_NODE(entry) ((struct f2fs_node *)(cache_address(entry))) >> + >> +static inline struct folio *cache_folio(const struct f2fs_cached_block *entry) >> +{ >> + return virt_to_folio(entry->data); >> +} >> + >> +int f2fs_init_cache(struct f2fs_sb_info *sbi, >> + struct f2fs_cached_block_list *cache, >> + enum f2fs_cache_type type); >> +void f2fs_destroy_cache(struct f2fs_cached_block_list *cache); >> +void f2fs_cache_get(struct f2fs_cached_block *entry); >> +struct f2fs_cached_block *f2fs_find_cache( >> + struct f2fs_cached_block_list *cache, >> + unsigned long index); >> +#define F2FS_CACHE_TAG_NONE 0 >> +#define F2FS_CACHE_TAG_DIRTY 1 >> +#define F2FS_CACHE_TAG_WRITEBACK 2 >> + >> +bool f2fs_trylock_cache(struct f2fs_cached_block *entry); >> +void f2fs_lock_cache(struct f2fs_cached_block *entry); >> +void f2fs_unlock_cache(struct f2fs_cached_block *entry); >> +bool f2fs_put_cache(struct f2fs_cached_block *entry, bool unlock); >> +bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry); >> +bool f2fs_clear_cache_dirty(struct f2fs_cached_block *entry); >> +void f2fs_drop_cache_dirty(struct f2fs_cached_block *entry); >> +void f2fs_force_clear_cache_dirty(struct f2fs_cached_block *entry); >> +void f2fs_start_cache_writeback(struct f2fs_cached_block *entry); >> +void f2fs_end_cache_writeback(struct f2fs_cached_block *entry); >> +unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache, >> + struct f2fs_cached_block **results, pgoff_t *first_index, >> + unsigned int max_items, int tag); >> +void f2fs_cache_gang_release(struct f2fs_cached_block **entries, >> + unsigned int nr_entries); >> +int f2fs_writeback_cache(struct f2fs_cached_block_list *cache, bool sync); >> +void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache); >> +void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, >> + enum page_type type); >> +void f2fs_cache_wait_writeback(struct f2fs_cached_block *entry); >> +struct f2fs_cached_block *f2fs_grab_cache(struct f2fs_cached_block_list *cache, >> + unsigned long index, int flags); >> +void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache, >> + unsigned long start, unsigned long len, bool drop_dirty); >> + >> +#endif /* _LINUX_F2FS_CACHE_H */ >> diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c >> index 6ae0eb37d20f..09474790035b 100644 >> --- a/fs/f2fs/data.c >> +++ b/fs/f2fs/data.c >> @@ -41,11 +41,6 @@ struct f2fs_folio_state { >> unsigned int read_pages_pending; >> }; >> >> -struct f2fs_bio { >> - struct work_struct work; >> - struct bio bio; >> -}; >> - >> #define F2FS_BIO_POOL_SIZE NR_CURSEG_TYPE >> >> int __init f2fs_init_bioset(void) >> @@ -69,7 +64,6 @@ bool f2fs_is_cp_guaranteed(const struct folio *folio) >> return folio_test_f2fs_gcing(fscrypt_pagecache_folio(folio)); >> >> inode = mapping->host; >> - sbi = F2FS_I_SB(inode); >> >> if (inode->i_ino == F2FS_META_INO(sbi) || >> inode->i_ino == F2FS_NODE_INO(sbi) || >> @@ -437,6 +431,65 @@ static void f2fs_write_end_io(struct bio *bio) >> } >> } >> >> +static void f2fs_cache_read_end_io(struct bio *bio) >> +{ >> + struct f2fs_cached_block *entry = F2FS_BIO(bio)->entry; >> + struct f2fs_sb_info *sbi = entry->cache->sbi; >> + enum count_type io_type = IS_META_CACHE(entry->cache) ? >> + F2FS_RD_META : F2FS_RD_NODE; >> + struct f2fs_cached_block *next; >> + >> + iostat_update_and_unbind_ctx(bio); >> + >> + if (time_to_inject(sbi, FAULT_READ_IO)) >> + bio->bi_status = BLK_STS_IOERR; >> + >> + while (entry) { >> + next = entry->next_entry; >> + entry->next_entry = NULL; >> + >> + if (bio->bi_status == BLK_STS_OK) >> + f2fs_cache_set_uptodate(entry); >> + >> + dec_page_count(sbi, io_type); >> + >> + f2fs_unlock_cache(entry); >> + entry = next; >> + } >> + bio_put(bio); >> +} >> + >> +static void f2fs_cache_write_end_io(struct bio *bio) >> +{ >> + struct f2fs_cached_block *entry = F2FS_BIO(bio)->entry; >> + struct f2fs_sb_info *sbi = entry->cache->sbi; >> + struct f2fs_cached_block *next; >> + >> + iostat_update_and_unbind_ctx(bio); >> + >> + if (time_to_inject(sbi, FAULT_WRITE_IO)) >> + bio->bi_status = BLK_STS_IOERR; >> + >> + if (bio->bi_status != BLK_STS_OK) >> + f2fs_stop_checkpoint(sbi, true, >> + STOP_CP_REASON_WRITE_FAIL); >> + >> + while (entry) { >> + next = entry->next_entry; >> + entry->next_entry = NULL; >> + >> + dec_page_count(sbi, F2FS_WB_CP_DATA); >> + >> + if (!get_pages(sbi, F2FS_WB_CP_DATA) && >> + wq_has_sleeper(&sbi->cp_wait)) >> + wake_up(&sbi->cp_wait); >> + >> + f2fs_end_cache_writeback(entry); >> + entry = next; >> + } >> + bio_put(bio); >> +} >> + >> #ifdef CONFIG_BLK_DEV_ZONED >> static void f2fs_zone_write_end_io(struct bio *bio) >> { >> @@ -444,7 +497,10 @@ static void f2fs_zone_write_end_io(struct bio *bio) >> >> bio->bi_private = io->bi_private; >> complete(&io->zone_wait); >> - f2fs_write_end_io(bio); >> + if (io->fio.is_cache) >> + f2fs_cache_write_end_io(bio); >> + else >> + f2fs_write_end_io(bio); >> } >> #endif >> >> @@ -531,12 +587,21 @@ static struct bio *__bio_alloc(struct f2fs_io_info *fio, int npages) >> fio->op | fio->op_flags | f2fs_io_flags(fio), >> GFP_NOIO, &f2fs_bioset); >> bio->bi_iter.bi_sector = sector; >> + F2FS_BIO(bio)->entry = NULL; >> + bio->bi_private = NULL; >> if (is_read_io(fio->op)) { >> - bio->bi_end_io = f2fs_read_end_io; >> - bio->bi_private = NULL; >> + if (fio->is_cache) >> + bio->bi_end_io = f2fs_cache_read_end_io; >> + else >> + bio->bi_end_io = f2fs_read_end_io; >> } else { >> - bio->bi_end_io = f2fs_write_end_io; >> - bio->bi_private = sbi; >> + if (fio->is_cache) { >> + bio->bi_end_io = f2fs_cache_write_end_io; >> + } else { >> + bio->bi_end_io = f2fs_write_end_io; >> + bio->bi_private = sbi; >> + } >> + >> bio->bi_write_hint = f2fs_io_type_to_rw_hint(sbi, >> fio->type, fio->temp); >> bio->bi_write_stream = f2fs_io_type_to_write_stream(bdev, fio->type, >> @@ -559,7 +624,7 @@ static void f2fs_set_bio_crypt_ctx(struct bio *bio, const struct inode *inode, >> * The f2fs garbage collector sets ->encrypted_page when it wants to >> * read/write raw data without encryption. >> */ >> - if (!fio || !fio->encrypted_page) >> + if (!fio || (!fio->encrypted_page && !fio->is_cache)) >> fscrypt_set_bio_crypt_ctx(bio, inode, >> (loff_t)first_idx << inode->i_blkbits, >> gfp_mask); >> @@ -769,6 +834,53 @@ void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, >> __submit_merged_write_cond(sbi, NULL, folio, 0, type, true); >> } >> >> +static bool __has_merged_cache(struct bio *bio, >> + struct f2fs_cached_block *target) >> +{ >> + struct f2fs_cached_block *entry; >> + >> + if (!bio) >> + return false; >> + >> + entry = F2FS_BIO(bio)->entry; >> + >> + while (entry) { >> + if (target && entry == target) >> + return true; >> + entry = entry->next_entry; >> + } >> + return false; >> +} >> + >> +bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, >> + enum page_type type) >> +{ >> + struct f2fs_sb_info *sbi = entry->cache->sbi; >> + enum temp_type temp; >> + bool ret = false; >> + >> + for (temp = HOT; temp < NR_TEMP_TYPE; temp++) { >> + enum page_type btype = PAGE_TYPE_OF_BIO(type); >> + struct f2fs_bio_info *io = sbi->write_io[btype] + temp; >> + struct f2fs_lock_context lc; >> + bool merged; >> + >> + f2fs_down_read_trace(&io->io_rwsem, &lc); >> + merged = __has_merged_cache(io->bio, entry); >> + f2fs_up_read_trace(&io->io_rwsem, &lc); >> + >> + if (merged) { >> + __f2fs_submit_merged_write(sbi, type, temp); >> + ret = true; >> + } >> + >> + /* TODO: use HOT temp only for meta pages now. */ >> + if (type >= META) >> + break; >> + } >> + return ret; >> +} >> + >> void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi) >> { >> f2fs_submit_merged_write(sbi, DATA); >> @@ -832,6 +944,8 @@ static bool io_type_is_mergeable(struct f2fs_bio_info *io, >> >> if (io->fio.op != fio->op) >> return false; >> + if (io->fio.is_cache != fio->is_cache) >> + return false; >> return (io->fio.op_flags & mask) == (fio->op_flags & mask); >> } >> >> @@ -1063,6 +1177,34 @@ static bool is_end_zone_blkaddr(struct f2fs_sb_info *sbi, block_t blkaddr) >> f2fs_blkz_is_seq(sbi, devi, blkaddr) && >> (blkaddr % sbi->blocks_per_blkz == sbi->blocks_per_blkz - 1); >> } >> + >> +static void f2fs_wait_zone_io_completion(struct f2fs_sb_info *sbi, >> + struct f2fs_bio_info *io, enum page_type btype) >> +{ >> + if (f2fs_sb_has_blkzoned(sbi) && btype < META && io->zone_pending_bio) { >> + wait_for_completion_io(&io->zone_wait); >> + bio_put(io->zone_pending_bio); >> + io->zone_pending_bio = NULL; >> + io->bi_private = NULL; >> + } >> +} >> + >> +static void f2fs_submit_zone_io(struct f2fs_sb_info *sbi, >> + struct f2fs_io_info *fio, struct f2fs_bio_info *io, >> + enum page_type btype) >> +{ >> + if (f2fs_sb_has_blkzoned(sbi) && btype < META && >> + is_end_zone_blkaddr(sbi, fio->new_blkaddr)) { >> + bio_get(io->bio); >> + reinit_completion(&io->zone_wait); >> + io->bi_private = io->bio->bi_private; >> + io->bio->bi_private = io; >> + io->bio->bi_end_io = f2fs_zone_write_end_io; >> + io->zone_pending_bio = io->bio; >> + __submit_merged_bio(io); >> + } >> + >> +} >> #endif >> >> void f2fs_submit_page_write(struct f2fs_io_info *fio) >> @@ -1079,14 +1221,8 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio) >> f2fs_down_write_trace(&io->io_rwsem, &lc); >> next: >> #ifdef CONFIG_BLK_DEV_ZONED >> - if (f2fs_sb_has_blkzoned(sbi) && btype < META && io->zone_pending_bio) { >> - wait_for_completion_io(&io->zone_wait); >> - bio_put(io->zone_pending_bio); >> - io->zone_pending_bio = NULL; >> - io->bi_private = NULL; >> - } >> + f2fs_wait_zone_io_completion(sbi, io, btype); >> #endif >> - >> if (fio->in_list) { >> spin_lock(&io->io_lock); >> if (list_empty(&io->io_list)) { >> @@ -1141,16 +1277,109 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio) >> >> trace_f2fs_submit_folio_write(fio->folio, fio); >> #ifdef CONFIG_BLK_DEV_ZONED >> - if (f2fs_sb_has_blkzoned(sbi) && btype < META && >> - is_end_zone_blkaddr(sbi, fio->new_blkaddr)) { >> - bio_get(io->bio); >> - reinit_completion(&io->zone_wait); >> - io->bi_private = io->bio->bi_private; >> - io->bio->bi_private = io; >> - io->bio->bi_end_io = f2fs_zone_write_end_io; >> - io->zone_pending_bio = io->bio; >> + f2fs_submit_zone_io(sbi, fio, io, btype); >> +#endif >> + >> + if (fio->in_list) >> + goto next; >> +out: >> + if (is_sbi_flag_set(sbi, SBI_IS_SHUTDOWN) || >> + !f2fs_is_checkpoint_ready(sbi)) >> __submit_merged_bio(io); >> + f2fs_up_write_trace(&io->io_rwsem, &lc); >> +} >> + >> +static void f2fs_bio_add_cache(struct f2fs_io_info *fio, struct bio *bio) >> +{ >> + struct f2fs_bio *fbio = F2FS_BIO(bio); >> + struct f2fs_cached_block *head = fbio->entry; >> + struct f2fs_cached_block *new = fio->cache_entry; >> + >> + new->next_entry = head; >> + fbio->entry = new; >> +} >> + >> +int f2fs_submit_cache_read(struct f2fs_io_info *fio) >> +{ >> + struct f2fs_sb_info *sbi = fio->sbi; >> + struct f2fs_cached_block *entry = fio->cache_entry; >> + struct bio *bio; >> + enum count_type io_type = IS_META_CACHE(entry->cache) ? >> + F2FS_RD_META : F2FS_RD_NODE; >> + >> + if (!f2fs_is_valid_blkaddr(fio->sbi, fio->new_blkaddr, >> + fio->is_por ? META_POR : (__is_meta_io(fio) ? >> + META_GENERIC : DATA_GENERIC_ENHANCE))) >> + return -EFSCORRUPTED; >> + >> + bio = __bio_alloc(fio, 1); >> + >> + bio_add_virt_nofail(bio, cache_address(entry), sbi->blocksize); >> + f2fs_bio_add_cache(fio, bio); >> + inc_page_count(sbi, io_type); >> + >> + f2fs_submit_read_bio(sbi, bio, fio->type); >> + return 0; >> +} >> + >> +void f2fs_submit_cache_write(struct f2fs_io_info *fio) >> +{ >> + struct f2fs_sb_info *sbi = fio->sbi; >> + enum page_type btype = PAGE_TYPE_OF_BIO(fio->type); >> + struct f2fs_bio_info *io = sbi->write_io[btype] + fio->temp; >> + struct f2fs_lock_context lc; >> + struct folio *folio; >> + >> + f2fs_bug_on(sbi, is_read_io(fio->op)); >> + >> + f2fs_down_write_trace(&io->io_rwsem, &lc); >> +next: >> +#ifdef CONFIG_BLK_DEV_ZONED >> + f2fs_wait_zone_io_completion(sbi, io, btype); >> +#endif >> + if (fio->in_list) { >> + spin_lock(&io->io_lock); >> + if (list_empty(&io->io_list)) { >> + spin_unlock(&io->io_lock); >> + goto out; >> + } >> + fio = list_first_entry(&io->io_list, >> + struct f2fs_io_info, list); >> + list_del(&fio->list); >> + spin_unlock(&io->io_lock); >> } >> + >> + verify_fio_blkaddr(fio); >> + >> + fio->submitted = 1; >> + inc_page_count(sbi, F2FS_WB_CP_DATA); >> + >> + if (io->bio && >> + (!io_is_mergeable(sbi, io->bio, io, fio, io->last_block_in_bio, >> + fio->new_blkaddr))) >> + __submit_merged_bio(io); >> +alloc_new: >> + if (io->bio == NULL) { >> + io->bio = __bio_alloc(fio, BIO_MAX_VECS); >> + io->fio = *fio; >> + } >> + >> + folio = cache_folio(fio->cache_entry); >> + >> + if (!bio_add_folio(io->bio, folio, sbi->blocksize, >> + offset_in_folio(folio, cache_address(fio->cache_entry)))) { >> + f2fs_bug_on(sbi, !F2FS_BIO(io->bio)->entry); >> + >> + __submit_merged_bio(io); >> + goto alloc_new; >> + } >> + >> + f2fs_bio_add_cache(fio, io->bio); >> + >> + io->last_block_in_bio = fio->new_blkaddr; >> + >> +#ifdef CONFIG_BLK_DEV_ZONED >> + f2fs_submit_zone_io(sbi, fio, io, btype); >> #endif >> if (fio->in_list) >> goto next; >> diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h >> index 1b96d8718c5c..8413983ea9d5 100644 >> --- a/fs/f2fs/f2fs.h >> +++ b/fs/f2fs/f2fs.h >> @@ -221,6 +221,8 @@ struct f2fs_rwsem { >> #endif >> }; >> >> +#include "cache.h" >> + >> struct f2fs_mount_info { >> unsigned long long opt; >> block_t root_reserved_blocks; /* root reserved blocks */ >> @@ -1370,8 +1372,10 @@ struct f2fs_io_info { >> unsigned int is_por:1; /* indicate IO is from recovery or not */ >> unsigned int encrypted:1; /* indicate file is encrypted */ >> unsigned int meta_gc:1; /* require meta inode GC */ >> + unsigned int is_cache:1; /* indicate IO is from internal cache */ >> enum iostat_type io_type; /* io type */ >> struct writeback_control *io_wbc; /* writeback control */ >> + struct f2fs_cached_block *cache_entry; >> struct bio **bio; /* bio for ipu */ >> sector_t *last_block; /* last block number in bio */ >> }; >> @@ -1781,6 +1785,12 @@ struct f2fs_gc_kthread { >> unsigned int boost_gc_greedy; >> }; >> >> +struct f2fs_bio { >> + struct work_struct work; >> + struct f2fs_cached_block *entry; >> + struct bio bio; >> +}; >> + >> struct f2fs_sb_info { >> struct super_block *sb; /* pointer to VFS super block */ >> struct proc_dir_entry *s_proc; /* proc entry */ >> @@ -2323,6 +2333,16 @@ static inline bool is_node_folio(struct folio *folio) >> return folio->mapping == NODE_MAPPING(F2FS_F_SB(folio)); >> } >> >> +static inline struct f2fs_bio *F2FS_BIO(struct bio *bio) >> +{ >> + return container_of(bio, struct f2fs_bio, bio); >> +} >> + >> +static inline bool f2fs_is_cache_bio(struct bio *bio) >> +{ >> + return F2FS_BIO(bio)->entry != NULL; >> +} >> + >> static inline bool is_sbi_flag_set(struct f2fs_sb_info *sbi, unsigned int type) >> { >> return test_bit(type, &sbi->s_flag); >> @@ -4237,6 +4257,8 @@ void f2fs_submit_merged_write_cond(struct f2fs_sb_info *sbi, >> nid_t ino, enum page_type type); >> void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, >> struct folio *folio, enum page_type type); >> +bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, >> + enum page_type type); >> void f2fs_submit_merged_ipu_write(struct f2fs_sb_info *sbi, >> struct bio **bio, struct folio *folio); >> void f2fs_submit_all_merged_ipu_writes(struct f2fs_sb_info *sbi); >> @@ -4244,6 +4266,8 @@ void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi); >> int f2fs_submit_page_bio(struct f2fs_io_info *fio); >> int f2fs_merge_page_bio(struct f2fs_io_info *fio); >> void f2fs_submit_page_write(struct f2fs_io_info *fio); >> +int f2fs_submit_cache_read(struct f2fs_io_info *fio); >> +void f2fs_submit_cache_write(struct f2fs_io_info *fio); >> struct block_device *f2fs_target_device(struct f2fs_sb_info *sbi, >> block_t blk_addr, sector_t *sector); >> int f2fs_target_device_index(struct f2fs_sb_info *sbi, block_t blkaddr); >> diff --git a/fs/f2fs/iostat.h b/fs/f2fs/iostat.h >> index 2025225b5bed..61c6bc8e3119 100644 >> --- a/fs/f2fs/iostat.h >> +++ b/fs/f2fs/iostat.h >> @@ -60,6 +60,13 @@ static inline struct bio_post_read_ctx *get_post_read_ctx(struct bio *bio) >> return iostat_ctx->post_read_ctx; >> } >> >> +static inline void iostat_set_post_read_ctx(struct bio *bio, void *ctx) >> +{ >> + struct bio_iostat_ctx *iostat_ctx = bio->bi_private; >> + >> + iostat_ctx->post_read_ctx = ctx; >> +} >> + >> extern void iostat_update_and_unbind_ctx(struct bio *bio); >> extern void iostat_alloc_and_bind_ctx(struct f2fs_sb_info *sbi, >> struct bio *bio, struct bio_post_read_ctx *ctx); >> @@ -81,6 +88,10 @@ static inline struct bio_post_read_ctx *get_post_read_ctx(struct bio *bio) >> { >> return bio->bi_private; >> } >> +static inline void iostat_set_post_read_ctx(struct bio *bio, void *ctx) >> +{ >> + bio->bi_private = ctx; >> +} >> static inline int f2fs_init_iostat_processing(void) { return 0; } >> static inline void f2fs_destroy_iostat_processing(void) {} >> static inline int f2fs_init_iostat(struct f2fs_sb_info *sbi) { return 0; } >> -- >> 2.49.0 >> >> >> >> _______________________________________________ >> Linux-f2fs-devel mailing list >> Linux-f2fs-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH v1 02/12] f2fs: cache: initialize meta cache 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel -1 siblings, 0 replies; 34+ messages in thread From: Chao Yu @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-f2fs-devel, linux-kernel, Chao Yu This patch introduces meta_blocks in f2fs_sb_info structure, initializes and destroys the meta cache during filesystem mount and unmount. It also introduces helper wrappers for meta cache operation. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.h | 12 ++++++++++++ fs/f2fs/data.c | 3 +++ fs/f2fs/f2fs.h | 3 +++ fs/f2fs/super.c | 10 +++++++++- 4 files changed, 27 insertions(+), 1 deletion(-) diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index 686a974008ae..3cd2abafe364 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -176,4 +176,16 @@ struct f2fs_cached_block *f2fs_grab_cache(struct f2fs_cached_block_list *cache, void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache, unsigned long start, unsigned long len, bool drop_dirty); +int f2fs_start_cache_wb_thread(struct f2fs_sb_info *sbi); +void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); + +#define META_CACHE(sbi) (&(sbi)->meta_blocks) + +#define f2fs_find_meta_cache(sbi, blkaddr) \ + f2fs_find_cache(META_CACHE(sbi), blkaddr) +#define f2fs_invalidate_meta_caches(sbi, start, len) \ + f2fs_drop_cache_range(META_CACHE(sbi), start, len, false) +#define f2fs_truncate_meta_caches(sbi, start, len) \ + f2fs_drop_cache_range(META_CACHE(sbi), start, len, true) + #endif /* _LINUX_F2FS_CACHE_H */ diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index 09474790035b..110282bb8dcd 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -695,6 +695,9 @@ static bool __has_merged_page(struct bio *bio, struct inode *inode, if (!inode && !folio && !ino) return true; + if (f2fs_is_cache_bio(bio)) + return false; + bio_for_each_folio_all(fi, bio) { struct folio *target = fi.folio; diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 8413983ea9d5..9b7c60bbb137 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -2105,6 +2105,9 @@ struct f2fs_sb_info { #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lock_class_key cp_global_sem_key; #endif + + /* f2fs internal cache */ + struct f2fs_cached_block_list meta_blocks; }; /* Definitions to access f2fs_sb_info */ diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index 3bdb0f891c35..bcfb1f97850e 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -2053,6 +2053,8 @@ static void f2fs_put_super(struct super_block *sb) iput(sbi->meta_inode); sbi->meta_inode = NULL; + f2fs_destroy_cache(META_CACHE(sbi)); + /* Should check the page counts after dropping all node/meta pages */ for (i = 0; i < NR_COUNT_TYPE; i++) { if (!get_pages(sbi, i)) @@ -5197,12 +5199,16 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) if (err) goto free_percpu; + err = f2fs_init_cache(sbi, META_CACHE(sbi), F2FS_META_CACHE); + if (err) + goto free_page_array_cache; + /* get an inode for meta space */ sbi->meta_inode = f2fs_iget(sb, F2FS_META_INO(sbi)); if (IS_ERR(sbi->meta_inode)) { f2fs_err(sbi, "Failed to read F2FS meta data inode"); err = PTR_ERR(sbi->meta_inode); - goto free_page_array_cache; + goto free_meta_cache; } err = f2fs_get_valid_checkpoint(sbi); @@ -5528,6 +5534,8 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) make_bad_inode(sbi->meta_inode); iput(sbi->meta_inode); sbi->meta_inode = NULL; +free_meta_cache: + f2fs_destroy_cache(META_CACHE(sbi)); free_page_array_cache: f2fs_destroy_page_array_cache(sbi); free_percpu: -- 2.49.0 ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [f2fs-dev] [PATCH v1 02/12] f2fs: cache: initialize meta cache @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-kernel, linux-f2fs-devel This patch introduces meta_blocks in f2fs_sb_info structure, initializes and destroys the meta cache during filesystem mount and unmount. It also introduces helper wrappers for meta cache operation. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.h | 12 ++++++++++++ fs/f2fs/data.c | 3 +++ fs/f2fs/f2fs.h | 3 +++ fs/f2fs/super.c | 10 +++++++++- 4 files changed, 27 insertions(+), 1 deletion(-) diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index 686a974008ae..3cd2abafe364 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -176,4 +176,16 @@ struct f2fs_cached_block *f2fs_grab_cache(struct f2fs_cached_block_list *cache, void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache, unsigned long start, unsigned long len, bool drop_dirty); +int f2fs_start_cache_wb_thread(struct f2fs_sb_info *sbi); +void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); + +#define META_CACHE(sbi) (&(sbi)->meta_blocks) + +#define f2fs_find_meta_cache(sbi, blkaddr) \ + f2fs_find_cache(META_CACHE(sbi), blkaddr) +#define f2fs_invalidate_meta_caches(sbi, start, len) \ + f2fs_drop_cache_range(META_CACHE(sbi), start, len, false) +#define f2fs_truncate_meta_caches(sbi, start, len) \ + f2fs_drop_cache_range(META_CACHE(sbi), start, len, true) + #endif /* _LINUX_F2FS_CACHE_H */ diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index 09474790035b..110282bb8dcd 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -695,6 +695,9 @@ static bool __has_merged_page(struct bio *bio, struct inode *inode, if (!inode && !folio && !ino) return true; + if (f2fs_is_cache_bio(bio)) + return false; + bio_for_each_folio_all(fi, bio) { struct folio *target = fi.folio; diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 8413983ea9d5..9b7c60bbb137 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -2105,6 +2105,9 @@ struct f2fs_sb_info { #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lock_class_key cp_global_sem_key; #endif + + /* f2fs internal cache */ + struct f2fs_cached_block_list meta_blocks; }; /* Definitions to access f2fs_sb_info */ diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index 3bdb0f891c35..bcfb1f97850e 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -2053,6 +2053,8 @@ static void f2fs_put_super(struct super_block *sb) iput(sbi->meta_inode); sbi->meta_inode = NULL; + f2fs_destroy_cache(META_CACHE(sbi)); + /* Should check the page counts after dropping all node/meta pages */ for (i = 0; i < NR_COUNT_TYPE; i++) { if (!get_pages(sbi, i)) @@ -5197,12 +5199,16 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) if (err) goto free_percpu; + err = f2fs_init_cache(sbi, META_CACHE(sbi), F2FS_META_CACHE); + if (err) + goto free_page_array_cache; + /* get an inode for meta space */ sbi->meta_inode = f2fs_iget(sb, F2FS_META_INO(sbi)); if (IS_ERR(sbi->meta_inode)) { f2fs_err(sbi, "Failed to read F2FS meta data inode"); err = PTR_ERR(sbi->meta_inode); - goto free_page_array_cache; + goto free_meta_cache; } err = f2fs_get_valid_checkpoint(sbi); @@ -5528,6 +5534,8 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) make_bad_inode(sbi->meta_inode); iput(sbi->meta_inode); sbi->meta_inode = NULL; +free_meta_cache: + f2fs_destroy_cache(META_CACHE(sbi)); free_page_array_cache: f2fs_destroy_page_array_cache(sbi); free_percpu: -- 2.49.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [PATCH v1 03/12] f2fs: cache: introduce shrinker 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel -1 siblings, 0 replies; 34+ messages in thread From: Chao Yu @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-f2fs-devel, linux-kernel, Chao Yu This patch integrates the metadata cache into the F2FS memory shrinker subsystem to reclaim clean, unreferenced cached blocks under memory pressure. It implements f2fs_shrink_cache() using a 3-phase cache reclamin method: 1. isolate clean entries from lru list 2. truncate from radix tree under lock 3. splice un-reclaimed entries back And hooks the new interface into f2fs_shrink_count() and f2fs_shrink_scan(). Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++ fs/f2fs/cache.h | 3 ++ fs/f2fs/shrinker.c | 12 +++++++ 3 files changed, 97 insertions(+) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index 08bc658166f7..c071364822c4 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -529,3 +529,85 @@ void f2fs_destroy_cache(struct f2fs_cached_block_list *cache) f2fs_put_cache(entry, true); goto next; } + +static unsigned long f2fs_do_shrink_cache(struct f2fs_cached_block_list *cache, + unsigned long nr_to_scan) +{ + struct f2fs_cached_block *entry, *next; + LIST_HEAD(dispose_list); + LIST_HEAD(keep_list); + unsigned long freed = 0; + unsigned long isolated = 0; + + /* Phase 1: Isolate candidate entries from LRU list into dispose_list */ + spin_lock(&cache->list_lock); + list_for_each_entry_safe(entry, next, &cache->lru_list, list) { + if (isolated >= nr_to_scan) + break; + + if (f2fs_cache_test_dirty(entry) || + f2fs_cache_test_writeback(entry) || + f2fs_cache_test_locked(entry)) + continue; + + if (f2fs_cache_refcount(entry) != 1) + continue; + + list_move_tail(&entry->list, &dispose_list); + isolated++; + } + spin_unlock(&cache->list_lock); + + /* Phase 2: Process isolated candidates one by one */ + while (1) { + spin_lock(&cache->list_lock); + entry = list_first_entry_or_null(&dispose_list, + struct f2fs_cached_block, list); + if (!entry) { + spin_unlock(&cache->list_lock); + break; + } + f2fs_cache_get(entry); + list_move_tail(&entry->list, &keep_list); + spin_unlock(&cache->list_lock); + + if (!f2fs_trylock_cache(entry)) { + f2fs_put_cache(entry, false); + continue; + } + + /* the entry has been truncated */ + if (!entry->cache) { + f2fs_put_cache(entry, true); + continue; + } + /* + * at least there are shrinker, radix tree and another user + * has referenced the entry. + */ + if (f2fs_cache_refcount(entry) >= 3) { + f2fs_put_cache(entry, true); + continue; + } + + f2fs_do_truncate_cache(entry, false); + + if (f2fs_put_cache(entry, true)) + freed++; + } + + /* Phase 3: Splice un-reclaimed entries back onto cache->lru_list */ + if (!list_empty(&keep_list)) { + spin_lock(&cache->list_lock); + list_splice_tail(&keep_list, &cache->lru_list); + spin_unlock(&cache->list_lock); + } + + return freed; +} + +unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, + unsigned long nr_to_scan) +{ + return f2fs_do_shrink_cache(META_CACHE(sbi), nr_to_scan); +} diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index 3cd2abafe364..d92b6f3ed585 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -188,4 +188,7 @@ void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); #define f2fs_truncate_meta_caches(sbi, start, len) \ f2fs_drop_cache_range(META_CACHE(sbi), start, len, true) +unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, + unsigned long nr_to_scan); + #endif /* _LINUX_F2FS_CACHE_H */ diff --git a/fs/f2fs/shrinker.c b/fs/f2fs/shrinker.c index 4f6bf5926de4..1755c85849e4 100644 --- a/fs/f2fs/shrinker.c +++ b/fs/f2fs/shrinker.c @@ -37,6 +37,11 @@ static unsigned long __count_extent_cache(struct f2fs_sb_info *sbi, atomic_read(&eti->total_ext_node); } +static unsigned long __count_cache(struct f2fs_sb_info *sbi) +{ + return sbi->meta_blocks.num_entries; +} + unsigned long f2fs_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { @@ -68,6 +73,9 @@ unsigned long f2fs_shrink_count(struct shrinker *shrink, /* count free nids cache entries */ count += __count_free_nids(sbi); + /* count generic cache entries */ + count += __count_cache(sbi); + spin_lock(&f2fs_list_lock); p = p->next; mutex_unlock(&sbi->umount_mutex); @@ -120,6 +128,10 @@ unsigned long f2fs_shrink_scan(struct shrinker *shrink, if (freed < nr) freed += f2fs_try_to_free_nids(sbi, nr - freed); + /* shrink generic cache entries */ + if (freed < nr) + freed += f2fs_shrink_cache(sbi, nr - freed); + spin_lock(&f2fs_list_lock); p = p->next; list_move_tail(&sbi->s_list, &f2fs_list); -- 2.49.0 ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [f2fs-dev] [PATCH v1 03/12] f2fs: cache: introduce shrinker @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-kernel, linux-f2fs-devel This patch integrates the metadata cache into the F2FS memory shrinker subsystem to reclaim clean, unreferenced cached blocks under memory pressure. It implements f2fs_shrink_cache() using a 3-phase cache reclamin method: 1. isolate clean entries from lru list 2. truncate from radix tree under lock 3. splice un-reclaimed entries back And hooks the new interface into f2fs_shrink_count() and f2fs_shrink_scan(). Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++ fs/f2fs/cache.h | 3 ++ fs/f2fs/shrinker.c | 12 +++++++ 3 files changed, 97 insertions(+) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index 08bc658166f7..c071364822c4 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -529,3 +529,85 @@ void f2fs_destroy_cache(struct f2fs_cached_block_list *cache) f2fs_put_cache(entry, true); goto next; } + +static unsigned long f2fs_do_shrink_cache(struct f2fs_cached_block_list *cache, + unsigned long nr_to_scan) +{ + struct f2fs_cached_block *entry, *next; + LIST_HEAD(dispose_list); + LIST_HEAD(keep_list); + unsigned long freed = 0; + unsigned long isolated = 0; + + /* Phase 1: Isolate candidate entries from LRU list into dispose_list */ + spin_lock(&cache->list_lock); + list_for_each_entry_safe(entry, next, &cache->lru_list, list) { + if (isolated >= nr_to_scan) + break; + + if (f2fs_cache_test_dirty(entry) || + f2fs_cache_test_writeback(entry) || + f2fs_cache_test_locked(entry)) + continue; + + if (f2fs_cache_refcount(entry) != 1) + continue; + + list_move_tail(&entry->list, &dispose_list); + isolated++; + } + spin_unlock(&cache->list_lock); + + /* Phase 2: Process isolated candidates one by one */ + while (1) { + spin_lock(&cache->list_lock); + entry = list_first_entry_or_null(&dispose_list, + struct f2fs_cached_block, list); + if (!entry) { + spin_unlock(&cache->list_lock); + break; + } + f2fs_cache_get(entry); + list_move_tail(&entry->list, &keep_list); + spin_unlock(&cache->list_lock); + + if (!f2fs_trylock_cache(entry)) { + f2fs_put_cache(entry, false); + continue; + } + + /* the entry has been truncated */ + if (!entry->cache) { + f2fs_put_cache(entry, true); + continue; + } + /* + * at least there are shrinker, radix tree and another user + * has referenced the entry. + */ + if (f2fs_cache_refcount(entry) >= 3) { + f2fs_put_cache(entry, true); + continue; + } + + f2fs_do_truncate_cache(entry, false); + + if (f2fs_put_cache(entry, true)) + freed++; + } + + /* Phase 3: Splice un-reclaimed entries back onto cache->lru_list */ + if (!list_empty(&keep_list)) { + spin_lock(&cache->list_lock); + list_splice_tail(&keep_list, &cache->lru_list); + spin_unlock(&cache->list_lock); + } + + return freed; +} + +unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, + unsigned long nr_to_scan) +{ + return f2fs_do_shrink_cache(META_CACHE(sbi), nr_to_scan); +} diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index 3cd2abafe364..d92b6f3ed585 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -188,4 +188,7 @@ void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); #define f2fs_truncate_meta_caches(sbi, start, len) \ f2fs_drop_cache_range(META_CACHE(sbi), start, len, true) +unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, + unsigned long nr_to_scan); + #endif /* _LINUX_F2FS_CACHE_H */ diff --git a/fs/f2fs/shrinker.c b/fs/f2fs/shrinker.c index 4f6bf5926de4..1755c85849e4 100644 --- a/fs/f2fs/shrinker.c +++ b/fs/f2fs/shrinker.c @@ -37,6 +37,11 @@ static unsigned long __count_extent_cache(struct f2fs_sb_info *sbi, atomic_read(&eti->total_ext_node); } +static unsigned long __count_cache(struct f2fs_sb_info *sbi) +{ + return sbi->meta_blocks.num_entries; +} + unsigned long f2fs_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { @@ -68,6 +73,9 @@ unsigned long f2fs_shrink_count(struct shrinker *shrink, /* count free nids cache entries */ count += __count_free_nids(sbi); + /* count generic cache entries */ + count += __count_cache(sbi); + spin_lock(&f2fs_list_lock); p = p->next; mutex_unlock(&sbi->umount_mutex); @@ -120,6 +128,10 @@ unsigned long f2fs_shrink_scan(struct shrinker *shrink, if (freed < nr) freed += f2fs_try_to_free_nids(sbi, nr - freed); + /* shrink generic cache entries */ + if (freed < nr) + freed += f2fs_shrink_cache(sbi, nr - freed); + spin_lock(&f2fs_list_lock); p = p->next; list_move_tail(&sbi->s_list, &f2fs_list); -- 2.49.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [PATCH v1 04/12] f2fs: cache: introduce writeback thread 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel -1 siblings, 0 replies; 34+ messages in thread From: Chao Yu @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-f2fs-devel, linux-kernel, Chao Yu This patch introduces a background writeback kthread (f2fs_writeback-x:y) to periodically flush dirty metadata cache entries with a default interval of 5 seconds. It manages thread lifecycle across mount, unmount, and remount (rw/ro) transitions, and hooks synchronous flushing into checkpoint commits. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 54 ++++++++++++++++++++++++++++++++++++++++++++ fs/f2fs/cache.h | 13 +++++++++++ fs/f2fs/checkpoint.c | 1 + fs/f2fs/f2fs.h | 3 +++ fs/f2fs/super.c | 31 ++++++++++++++++++++++++- 5 files changed, 101 insertions(+), 1 deletion(-) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index c071364822c4..cb5b26046162 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -611,3 +611,57 @@ unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, { return f2fs_do_shrink_cache(META_CACHE(sbi), nr_to_scan); } + +static int f2fs_cache_writeback_kthread(void *data) +{ + struct f2fs_sb_info *sbi = data; + struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread; + wait_queue_head_t *wq = &cache_thread->cache_wb_wq; + unsigned int interval = DEF_DIRTY_CACHE_TIMEOUT; + + set_freezable(); + + while (!kthread_should_stop()) { + wait_event_freezable_timeout(*wq, + kthread_should_stop() || + cache_thread->cache_wb_task == NULL, + msecs_to_jiffies(interval)); + + if (kthread_should_stop()) + break; + if (f2fs_cp_error(sbi)) + continue; + } + return 0; +} + +int f2fs_start_cache_wb_thread(struct f2fs_sb_info *sbi) +{ + struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread; + dev_t dev = sbi->sb->s_dev; + char name[36]; + + if (cache_thread->cache_wb_task) + return 0; + + init_waitqueue_head(&cache_thread->cache_wb_wq); + snprintf(name, sizeof(name), "f2fs_writeback-%u:%u", + MAJOR(dev), MINOR(dev)); + + cache_thread->cache_wb_task = kthread_run(f2fs_cache_writeback_kthread, + sbi, "%s", name); + if (IS_ERR(cache_thread->cache_wb_task)) + return PTR_ERR(cache_thread->cache_wb_task); + return 0; +} + +void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi) +{ + struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread; + + if (!cache_thread->cache_wb_task) + return; + + kthread_stop(cache_thread->cache_wb_task); + cache_thread->cache_wb_task = NULL; +} diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index d92b6f3ed585..5e9fe8c0b15c 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -191,4 +191,17 @@ void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, unsigned long nr_to_scan); +#define DEF_DIRTY_CACHE_TIMEOUT 5000 + +struct f2fs_cache_kthread { + struct task_struct *cache_wb_task; + wait_queue_head_t cache_wb_wq; + atomic_t cache_wb_trigger; + unsigned int cache_wb_interval_ms; +}; + +int f2fs_start_cache_wb_thread(struct f2fs_sb_info *sbi); +void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); +void f2fs_sync_cache_wb(struct f2fs_sb_info *sbi); + #endif /* _LINUX_F2FS_CACHE_H */ diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c index f81ba8cc861a..729d19680caf 100644 --- a/fs/f2fs/checkpoint.c +++ b/fs/f2fs/checkpoint.c @@ -1596,6 +1596,7 @@ static int block_operations(struct f2fs_sb_info *sbi) * sbi->node_change is used only for AIO write_begin path which produces * dirty node blocks and some checkpoint values by block allocation. */ + __prepare_cp_block(sbi); f2fs_up_write(&sbi->node_change); return err; diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 9b7c60bbb137..6e20b3586f26 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -2108,6 +2108,9 @@ struct f2fs_sb_info { /* f2fs internal cache */ struct f2fs_cached_block_list meta_blocks; + + /* internal cache flush thread */ + struct f2fs_cache_kthread cache_thread; }; /* Definitions to access f2fs_sb_info */ diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index bcfb1f97850e..89affe72f4fc 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -1998,6 +1998,7 @@ static void f2fs_put_super(struct super_block *sb) * flush all issued checkpoints and stop checkpoint issue thread. * after then, all checkpoints should be done by each process context. */ + f2fs_stop_cache_wb_thread(sbi); f2fs_stop_ckpt_thread(sbi); /* @@ -2799,6 +2800,7 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) unsigned int flags = fc->sb_flags; int err; bool need_restart_gc = false, need_stop_gc = false; + bool need_restart_wb = false, need_stop_wb = false; bool need_restart_flush = false, need_stop_flush = false; bool need_restart_discard = false, need_stop_discard = false; bool need_enable_checkpoint = false, need_disable_checkpoint = false; @@ -2957,6 +2959,18 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) need_stop_gc = true; } + if (flags & SB_RDONLY) { + if (sbi->cache_thread.cache_wb_task) { + f2fs_stop_cache_wb_thread(sbi); + need_restart_wb = true; + } + } else if (!sbi->cache_thread.cache_wb_task) { + err = f2fs_start_cache_wb_thread(sbi); + if (err) + goto restore_gc; + need_stop_wb = true; + } + if (flags & SB_RDONLY) { sync_inodes_sb(sb); @@ -2979,7 +2993,7 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) } else { err = f2fs_create_flush_cmd_control(sbi); if (err) - goto restore_gc; + goto restore_wb; need_stop_flush = true; } @@ -3075,6 +3089,13 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) clear_opt(sbi, FLUSH_MERGE); f2fs_destroy_flush_cmd_control(sbi, false); } +restore_wb: + if (need_restart_wb) { + if (f2fs_start_cache_wb_thread(sbi)) + f2fs_warn(sbi, "background cache writeback thread has stopped"); + } else if (need_stop_wb) { + f2fs_stop_cache_wb_thread(sbi); + } restore_gc: if (need_restart_gc) { if (f2fs_start_gc_thread(sbi)) @@ -5461,6 +5482,12 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) goto sync_free_meta; } + if (!f2fs_readonly(sb)) { + err = f2fs_start_cache_wb_thread(sbi); + if (err) + goto stop_gc_thread; + } + /* recover broken superblock */ if (recovery) { err = f2fs_commit_super(sbi, true); @@ -5483,6 +5510,8 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) sbi->umount_lock_holder = NULL; return 0; +stop_gc_thread: + f2fs_stop_gc_thread(sbi); sync_free_meta: /* safe to flush all the data */ sync_filesystem(sbi->sb); -- 2.49.0 ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [f2fs-dev] [PATCH v1 04/12] f2fs: cache: introduce writeback thread @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-kernel, linux-f2fs-devel This patch introduces a background writeback kthread (f2fs_writeback-x:y) to periodically flush dirty metadata cache entries with a default interval of 5 seconds. It manages thread lifecycle across mount, unmount, and remount (rw/ro) transitions, and hooks synchronous flushing into checkpoint commits. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 54 ++++++++++++++++++++++++++++++++++++++++++++ fs/f2fs/cache.h | 13 +++++++++++ fs/f2fs/checkpoint.c | 1 + fs/f2fs/f2fs.h | 3 +++ fs/f2fs/super.c | 31 ++++++++++++++++++++++++- 5 files changed, 101 insertions(+), 1 deletion(-) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index c071364822c4..cb5b26046162 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -611,3 +611,57 @@ unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, { return f2fs_do_shrink_cache(META_CACHE(sbi), nr_to_scan); } + +static int f2fs_cache_writeback_kthread(void *data) +{ + struct f2fs_sb_info *sbi = data; + struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread; + wait_queue_head_t *wq = &cache_thread->cache_wb_wq; + unsigned int interval = DEF_DIRTY_CACHE_TIMEOUT; + + set_freezable(); + + while (!kthread_should_stop()) { + wait_event_freezable_timeout(*wq, + kthread_should_stop() || + cache_thread->cache_wb_task == NULL, + msecs_to_jiffies(interval)); + + if (kthread_should_stop()) + break; + if (f2fs_cp_error(sbi)) + continue; + } + return 0; +} + +int f2fs_start_cache_wb_thread(struct f2fs_sb_info *sbi) +{ + struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread; + dev_t dev = sbi->sb->s_dev; + char name[36]; + + if (cache_thread->cache_wb_task) + return 0; + + init_waitqueue_head(&cache_thread->cache_wb_wq); + snprintf(name, sizeof(name), "f2fs_writeback-%u:%u", + MAJOR(dev), MINOR(dev)); + + cache_thread->cache_wb_task = kthread_run(f2fs_cache_writeback_kthread, + sbi, "%s", name); + if (IS_ERR(cache_thread->cache_wb_task)) + return PTR_ERR(cache_thread->cache_wb_task); + return 0; +} + +void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi) +{ + struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread; + + if (!cache_thread->cache_wb_task) + return; + + kthread_stop(cache_thread->cache_wb_task); + cache_thread->cache_wb_task = NULL; +} diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index d92b6f3ed585..5e9fe8c0b15c 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -191,4 +191,17 @@ void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, unsigned long nr_to_scan); +#define DEF_DIRTY_CACHE_TIMEOUT 5000 + +struct f2fs_cache_kthread { + struct task_struct *cache_wb_task; + wait_queue_head_t cache_wb_wq; + atomic_t cache_wb_trigger; + unsigned int cache_wb_interval_ms; +}; + +int f2fs_start_cache_wb_thread(struct f2fs_sb_info *sbi); +void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); +void f2fs_sync_cache_wb(struct f2fs_sb_info *sbi); + #endif /* _LINUX_F2FS_CACHE_H */ diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c index f81ba8cc861a..729d19680caf 100644 --- a/fs/f2fs/checkpoint.c +++ b/fs/f2fs/checkpoint.c @@ -1596,6 +1596,7 @@ static int block_operations(struct f2fs_sb_info *sbi) * sbi->node_change is used only for AIO write_begin path which produces * dirty node blocks and some checkpoint values by block allocation. */ + __prepare_cp_block(sbi); f2fs_up_write(&sbi->node_change); return err; diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 9b7c60bbb137..6e20b3586f26 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -2108,6 +2108,9 @@ struct f2fs_sb_info { /* f2fs internal cache */ struct f2fs_cached_block_list meta_blocks; + + /* internal cache flush thread */ + struct f2fs_cache_kthread cache_thread; }; /* Definitions to access f2fs_sb_info */ diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index bcfb1f97850e..89affe72f4fc 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -1998,6 +1998,7 @@ static void f2fs_put_super(struct super_block *sb) * flush all issued checkpoints and stop checkpoint issue thread. * after then, all checkpoints should be done by each process context. */ + f2fs_stop_cache_wb_thread(sbi); f2fs_stop_ckpt_thread(sbi); /* @@ -2799,6 +2800,7 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) unsigned int flags = fc->sb_flags; int err; bool need_restart_gc = false, need_stop_gc = false; + bool need_restart_wb = false, need_stop_wb = false; bool need_restart_flush = false, need_stop_flush = false; bool need_restart_discard = false, need_stop_discard = false; bool need_enable_checkpoint = false, need_disable_checkpoint = false; @@ -2957,6 +2959,18 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) need_stop_gc = true; } + if (flags & SB_RDONLY) { + if (sbi->cache_thread.cache_wb_task) { + f2fs_stop_cache_wb_thread(sbi); + need_restart_wb = true; + } + } else if (!sbi->cache_thread.cache_wb_task) { + err = f2fs_start_cache_wb_thread(sbi); + if (err) + goto restore_gc; + need_stop_wb = true; + } + if (flags & SB_RDONLY) { sync_inodes_sb(sb); @@ -2979,7 +2993,7 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) } else { err = f2fs_create_flush_cmd_control(sbi); if (err) - goto restore_gc; + goto restore_wb; need_stop_flush = true; } @@ -3075,6 +3089,13 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) clear_opt(sbi, FLUSH_MERGE); f2fs_destroy_flush_cmd_control(sbi, false); } +restore_wb: + if (need_restart_wb) { + if (f2fs_start_cache_wb_thread(sbi)) + f2fs_warn(sbi, "background cache writeback thread has stopped"); + } else if (need_stop_wb) { + f2fs_stop_cache_wb_thread(sbi); + } restore_gc: if (need_restart_gc) { if (f2fs_start_gc_thread(sbi)) @@ -5461,6 +5482,12 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) goto sync_free_meta; } + if (!f2fs_readonly(sb)) { + err = f2fs_start_cache_wb_thread(sbi); + if (err) + goto stop_gc_thread; + } + /* recover broken superblock */ if (recovery) { err = f2fs_commit_super(sbi, true); @@ -5483,6 +5510,8 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) sbi->umount_lock_holder = NULL; return 0; +stop_gc_thread: + f2fs_stop_gc_thread(sbi); sync_free_meta: /* safe to flush all the data */ sync_filesystem(sbi->sb); -- 2.49.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: [f2fs-dev] [PATCH v1 04/12] f2fs: cache: introduce writeback thread 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel @ 2026-08-20 5:09 ` Jaegeuk Kim via Linux-f2fs-devel -1 siblings, 0 replies; 34+ messages in thread From: Jaegeuk Kim @ 2026-08-20 5:09 UTC (permalink / raw) To: Chao Yu; +Cc: linux-kernel, linux-f2fs-devel On 08/20, Chao Yu via Linux-f2fs-devel wrote: > This patch introduces a background writeback kthread (f2fs_writeback-x:y) > to periodically flush dirty metadata cache entries with a default > interval of 5 seconds. > > It manages thread lifecycle across mount, unmount, and remount (rw/ro) > transitions, and hooks synchronous flushing into checkpoint commits. > > Signed-off-by: Chao Yu <chao@kernel.org> > --- > fs/f2fs/cache.c | 54 ++++++++++++++++++++++++++++++++++++++++++++ > fs/f2fs/cache.h | 13 +++++++++++ > fs/f2fs/checkpoint.c | 1 + > fs/f2fs/f2fs.h | 3 +++ > fs/f2fs/super.c | 31 ++++++++++++++++++++++++- > 5 files changed, 101 insertions(+), 1 deletion(-) > > diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c > index c071364822c4..cb5b26046162 100644 > --- a/fs/f2fs/cache.c > +++ b/fs/f2fs/cache.c > @@ -611,3 +611,57 @@ unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, > { > return f2fs_do_shrink_cache(META_CACHE(sbi), nr_to_scan); > } > + > +static int f2fs_cache_writeback_kthread(void *data) > +{ > + struct f2fs_sb_info *sbi = data; > + struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread; > + wait_queue_head_t *wq = &cache_thread->cache_wb_wq; > + unsigned int interval = DEF_DIRTY_CACHE_TIMEOUT; > + > + set_freezable(); > + > + while (!kthread_should_stop()) { > + wait_event_freezable_timeout(*wq, > + kthread_should_stop() || > + cache_thread->cache_wb_task == NULL, > + msecs_to_jiffies(interval)); > + > + if (kthread_should_stop()) > + break; > + if (f2fs_cp_error(sbi)) > + continue; > + } > + return 0; > +} > + > +int f2fs_start_cache_wb_thread(struct f2fs_sb_info *sbi) > +{ > + struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread; > + dev_t dev = sbi->sb->s_dev; > + char name[36]; > + > + if (cache_thread->cache_wb_task) > + return 0; > + > + init_waitqueue_head(&cache_thread->cache_wb_wq); > + snprintf(name, sizeof(name), "f2fs_writeback-%u:%u", > + MAJOR(dev), MINOR(dev)); > + > + cache_thread->cache_wb_task = kthread_run(f2fs_cache_writeback_kthread, > + sbi, "%s", name); > + if (IS_ERR(cache_thread->cache_wb_task)) > + return PTR_ERR(cache_thread->cache_wb_task); > + return 0; > +} > + > +void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi) > +{ > + struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread; > + > + if (!cache_thread->cache_wb_task) > + return; > + > + kthread_stop(cache_thread->cache_wb_task); > + cache_thread->cache_wb_task = NULL; > +} > diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h > index d92b6f3ed585..5e9fe8c0b15c 100644 > --- a/fs/f2fs/cache.h > +++ b/fs/f2fs/cache.h > @@ -191,4 +191,17 @@ void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); > unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, > unsigned long nr_to_scan); > > +#define DEF_DIRTY_CACHE_TIMEOUT 5000 > + > +struct f2fs_cache_kthread { > + struct task_struct *cache_wb_task; > + wait_queue_head_t cache_wb_wq; > + atomic_t cache_wb_trigger; > + unsigned int cache_wb_interval_ms; > +}; > + > +int f2fs_start_cache_wb_thread(struct f2fs_sb_info *sbi); > +void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); > +void f2fs_sync_cache_wb(struct f2fs_sb_info *sbi); > + > #endif /* _LINUX_F2FS_CACHE_H */ > diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c > index f81ba8cc861a..729d19680caf 100644 > --- a/fs/f2fs/checkpoint.c > +++ b/fs/f2fs/checkpoint.c > @@ -1596,6 +1596,7 @@ static int block_operations(struct f2fs_sb_info *sbi) > * sbi->node_change is used only for AIO write_begin path which produces > * dirty node blocks and some checkpoint values by block allocation. > */ > + Unnecessary line. > __prepare_cp_block(sbi); > f2fs_up_write(&sbi->node_change); > return err; > diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h > index 9b7c60bbb137..6e20b3586f26 100644 > --- a/fs/f2fs/f2fs.h > +++ b/fs/f2fs/f2fs.h > @@ -2108,6 +2108,9 @@ struct f2fs_sb_info { > > /* f2fs internal cache */ > struct f2fs_cached_block_list meta_blocks; > + > + /* internal cache flush thread */ > + struct f2fs_cache_kthread cache_thread; > }; > > /* Definitions to access f2fs_sb_info */ > diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c > index bcfb1f97850e..89affe72f4fc 100644 > --- a/fs/f2fs/super.c > +++ b/fs/f2fs/super.c > @@ -1998,6 +1998,7 @@ static void f2fs_put_super(struct super_block *sb) > * flush all issued checkpoints and stop checkpoint issue thread. > * after then, all checkpoints should be done by each process context. > */ > + f2fs_stop_cache_wb_thread(sbi); > f2fs_stop_ckpt_thread(sbi); > > /* > @@ -2799,6 +2800,7 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) > unsigned int flags = fc->sb_flags; > int err; > bool need_restart_gc = false, need_stop_gc = false; > + bool need_restart_wb = false, need_stop_wb = false; > bool need_restart_flush = false, need_stop_flush = false; > bool need_restart_discard = false, need_stop_discard = false; > bool need_enable_checkpoint = false, need_disable_checkpoint = false; > @@ -2957,6 +2959,18 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) > need_stop_gc = true; > } > > + if (flags & SB_RDONLY) { > + if (sbi->cache_thread.cache_wb_task) { > + f2fs_stop_cache_wb_thread(sbi); > + need_restart_wb = true; > + } > + } else if (!sbi->cache_thread.cache_wb_task) { > + err = f2fs_start_cache_wb_thread(sbi); > + if (err) > + goto restore_gc; > + need_stop_wb = true; > + } > + > if (flags & SB_RDONLY) { > sync_inodes_sb(sb); > > @@ -2979,7 +2993,7 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) > } else { > err = f2fs_create_flush_cmd_control(sbi); > if (err) > - goto restore_gc; > + goto restore_wb; > need_stop_flush = true; > } > > @@ -3075,6 +3089,13 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) > clear_opt(sbi, FLUSH_MERGE); > f2fs_destroy_flush_cmd_control(sbi, false); > } > +restore_wb: > + if (need_restart_wb) { > + if (f2fs_start_cache_wb_thread(sbi)) > + f2fs_warn(sbi, "background cache writeback thread has stopped"); > + } else if (need_stop_wb) { > + f2fs_stop_cache_wb_thread(sbi); > + } > restore_gc: > if (need_restart_gc) { > if (f2fs_start_gc_thread(sbi)) > @@ -5461,6 +5482,12 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) > goto sync_free_meta; > } > > + if (!f2fs_readonly(sb)) { > + err = f2fs_start_cache_wb_thread(sbi); > + if (err) > + goto stop_gc_thread; > + } > + > /* recover broken superblock */ > if (recovery) { > err = f2fs_commit_super(sbi, true); > @@ -5483,6 +5510,8 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) > sbi->umount_lock_holder = NULL; > return 0; > > +stop_gc_thread: > + f2fs_stop_gc_thread(sbi); > sync_free_meta: > /* safe to flush all the data */ > sync_filesystem(sbi->sb); > -- > 2.49.0 > > > > _______________________________________________ > Linux-f2fs-devel mailing list > Linux-f2fs-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [f2fs-dev] [PATCH v1 04/12] f2fs: cache: introduce writeback thread @ 2026-08-20 5:09 ` Jaegeuk Kim via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Jaegeuk Kim via Linux-f2fs-devel @ 2026-08-20 5:09 UTC (permalink / raw) To: Chao Yu; +Cc: linux-kernel, linux-f2fs-devel On 08/20, Chao Yu via Linux-f2fs-devel wrote: > This patch introduces a background writeback kthread (f2fs_writeback-x:y) > to periodically flush dirty metadata cache entries with a default > interval of 5 seconds. > > It manages thread lifecycle across mount, unmount, and remount (rw/ro) > transitions, and hooks synchronous flushing into checkpoint commits. > > Signed-off-by: Chao Yu <chao@kernel.org> > --- > fs/f2fs/cache.c | 54 ++++++++++++++++++++++++++++++++++++++++++++ > fs/f2fs/cache.h | 13 +++++++++++ > fs/f2fs/checkpoint.c | 1 + > fs/f2fs/f2fs.h | 3 +++ > fs/f2fs/super.c | 31 ++++++++++++++++++++++++- > 5 files changed, 101 insertions(+), 1 deletion(-) > > diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c > index c071364822c4..cb5b26046162 100644 > --- a/fs/f2fs/cache.c > +++ b/fs/f2fs/cache.c > @@ -611,3 +611,57 @@ unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, > { > return f2fs_do_shrink_cache(META_CACHE(sbi), nr_to_scan); > } > + > +static int f2fs_cache_writeback_kthread(void *data) > +{ > + struct f2fs_sb_info *sbi = data; > + struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread; > + wait_queue_head_t *wq = &cache_thread->cache_wb_wq; > + unsigned int interval = DEF_DIRTY_CACHE_TIMEOUT; > + > + set_freezable(); > + > + while (!kthread_should_stop()) { > + wait_event_freezable_timeout(*wq, > + kthread_should_stop() || > + cache_thread->cache_wb_task == NULL, > + msecs_to_jiffies(interval)); > + > + if (kthread_should_stop()) > + break; > + if (f2fs_cp_error(sbi)) > + continue; > + } > + return 0; > +} > + > +int f2fs_start_cache_wb_thread(struct f2fs_sb_info *sbi) > +{ > + struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread; > + dev_t dev = sbi->sb->s_dev; > + char name[36]; > + > + if (cache_thread->cache_wb_task) > + return 0; > + > + init_waitqueue_head(&cache_thread->cache_wb_wq); > + snprintf(name, sizeof(name), "f2fs_writeback-%u:%u", > + MAJOR(dev), MINOR(dev)); > + > + cache_thread->cache_wb_task = kthread_run(f2fs_cache_writeback_kthread, > + sbi, "%s", name); > + if (IS_ERR(cache_thread->cache_wb_task)) > + return PTR_ERR(cache_thread->cache_wb_task); > + return 0; > +} > + > +void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi) > +{ > + struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread; > + > + if (!cache_thread->cache_wb_task) > + return; > + > + kthread_stop(cache_thread->cache_wb_task); > + cache_thread->cache_wb_task = NULL; > +} > diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h > index d92b6f3ed585..5e9fe8c0b15c 100644 > --- a/fs/f2fs/cache.h > +++ b/fs/f2fs/cache.h > @@ -191,4 +191,17 @@ void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); > unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, > unsigned long nr_to_scan); > > +#define DEF_DIRTY_CACHE_TIMEOUT 5000 > + > +struct f2fs_cache_kthread { > + struct task_struct *cache_wb_task; > + wait_queue_head_t cache_wb_wq; > + atomic_t cache_wb_trigger; > + unsigned int cache_wb_interval_ms; > +}; > + > +int f2fs_start_cache_wb_thread(struct f2fs_sb_info *sbi); > +void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); > +void f2fs_sync_cache_wb(struct f2fs_sb_info *sbi); > + > #endif /* _LINUX_F2FS_CACHE_H */ > diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c > index f81ba8cc861a..729d19680caf 100644 > --- a/fs/f2fs/checkpoint.c > +++ b/fs/f2fs/checkpoint.c > @@ -1596,6 +1596,7 @@ static int block_operations(struct f2fs_sb_info *sbi) > * sbi->node_change is used only for AIO write_begin path which produces > * dirty node blocks and some checkpoint values by block allocation. > */ > + Unnecessary line. > __prepare_cp_block(sbi); > f2fs_up_write(&sbi->node_change); > return err; > diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h > index 9b7c60bbb137..6e20b3586f26 100644 > --- a/fs/f2fs/f2fs.h > +++ b/fs/f2fs/f2fs.h > @@ -2108,6 +2108,9 @@ struct f2fs_sb_info { > > /* f2fs internal cache */ > struct f2fs_cached_block_list meta_blocks; > + > + /* internal cache flush thread */ > + struct f2fs_cache_kthread cache_thread; > }; > > /* Definitions to access f2fs_sb_info */ > diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c > index bcfb1f97850e..89affe72f4fc 100644 > --- a/fs/f2fs/super.c > +++ b/fs/f2fs/super.c > @@ -1998,6 +1998,7 @@ static void f2fs_put_super(struct super_block *sb) > * flush all issued checkpoints and stop checkpoint issue thread. > * after then, all checkpoints should be done by each process context. > */ > + f2fs_stop_cache_wb_thread(sbi); > f2fs_stop_ckpt_thread(sbi); > > /* > @@ -2799,6 +2800,7 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) > unsigned int flags = fc->sb_flags; > int err; > bool need_restart_gc = false, need_stop_gc = false; > + bool need_restart_wb = false, need_stop_wb = false; > bool need_restart_flush = false, need_stop_flush = false; > bool need_restart_discard = false, need_stop_discard = false; > bool need_enable_checkpoint = false, need_disable_checkpoint = false; > @@ -2957,6 +2959,18 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) > need_stop_gc = true; > } > > + if (flags & SB_RDONLY) { > + if (sbi->cache_thread.cache_wb_task) { > + f2fs_stop_cache_wb_thread(sbi); > + need_restart_wb = true; > + } > + } else if (!sbi->cache_thread.cache_wb_task) { > + err = f2fs_start_cache_wb_thread(sbi); > + if (err) > + goto restore_gc; > + need_stop_wb = true; > + } > + > if (flags & SB_RDONLY) { > sync_inodes_sb(sb); > > @@ -2979,7 +2993,7 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) > } else { > err = f2fs_create_flush_cmd_control(sbi); > if (err) > - goto restore_gc; > + goto restore_wb; > need_stop_flush = true; > } > > @@ -3075,6 +3089,13 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) > clear_opt(sbi, FLUSH_MERGE); > f2fs_destroy_flush_cmd_control(sbi, false); > } > +restore_wb: > + if (need_restart_wb) { > + if (f2fs_start_cache_wb_thread(sbi)) > + f2fs_warn(sbi, "background cache writeback thread has stopped"); > + } else if (need_stop_wb) { > + f2fs_stop_cache_wb_thread(sbi); > + } > restore_gc: > if (need_restart_gc) { > if (f2fs_start_gc_thread(sbi)) > @@ -5461,6 +5482,12 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) > goto sync_free_meta; > } > > + if (!f2fs_readonly(sb)) { > + err = f2fs_start_cache_wb_thread(sbi); > + if (err) > + goto stop_gc_thread; > + } > + > /* recover broken superblock */ > if (recovery) { > err = f2fs_commit_super(sbi, true); > @@ -5483,6 +5510,8 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) > sbi->umount_lock_holder = NULL; > return 0; > > +stop_gc_thread: > + f2fs_stop_gc_thread(sbi); > sync_free_meta: > /* safe to flush all the data */ > sync_filesystem(sbi->sb); > -- > 2.49.0 > > > > _______________________________________________ > Linux-f2fs-devel mailing list > Linux-f2fs-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply [flat|nested] 34+ messages in thread
* [f2fs-dev] [PATCH v1 05/12] f2fs: cache: use meta cache 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 ` Chao Yu -1 siblings, 0 replies; 34+ messages in thread From: Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-kernel, linux-f2fs-devel This patch migrates F2FS meta block caching from the fake VFS inode page cache (sbi->meta_inode) to meta cache (sbi->meta_blocks). It converts CP, SIT, NAT, SSA, recovery, and GC metadata I/O paths to operate on struct f2fs_cached_block instead of folio, and removes sbi->meta_inode. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 2 + fs/f2fs/checkpoint.c | 394 ++++++++++++++++++---------------------- fs/f2fs/compress.c | 4 +- fs/f2fs/data.c | 29 ++- fs/f2fs/debug.c | 12 +- fs/f2fs/f2fs.h | 81 +++------ fs/f2fs/file.c | 4 +- fs/f2fs/gc.c | 134 +++++++------- fs/f2fs/inline.c | 9 +- fs/f2fs/inode.c | 9 +- fs/f2fs/node.c | 125 +++++++------ fs/f2fs/node.h | 4 +- fs/f2fs/recovery.c | 198 ++++++++++---------- fs/f2fs/segment.c | 201 ++++++++++---------- fs/f2fs/segment.h | 31 +++- fs/f2fs/super.c | 31 +--- include/linux/f2fs_fs.h | 1 - 17 files changed, 606 insertions(+), 663 deletions(-) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index cb5b26046162..afef808e485a 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -631,6 +631,8 @@ static int f2fs_cache_writeback_kthread(void *data) break; if (f2fs_cp_error(sbi)) continue; + + f2fs_write_meta_caches(sbi); } return 0; } diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c index 729d19680caf..1a7083540b82 100644 --- a/fs/f2fs/checkpoint.c +++ b/fs/f2fs/checkpoint.c @@ -17,6 +17,7 @@ #include <linux/delayacct.h> #include <linux/ioprio.h> #include <linux/math64.h> +#include <linux/freezer.h> #include "f2fs.h" #include "node.h" @@ -235,27 +236,27 @@ struct kmem_cache *f2fs_inode_entry_slab; /* * We guarantee no failure on the returned page. */ -struct folio *f2fs_grab_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index) +struct f2fs_cached_block *f2fs_grab_meta_cache(struct f2fs_sb_info *sbi, + pgoff_t index) { - struct address_space *mapping = META_MAPPING(sbi); - struct folio *folio; + struct f2fs_cached_block *entry; repeat: - folio = f2fs_grab_cache_folio(mapping, index, false); - if (IS_ERR(folio)) { + entry = f2fs_grab_cache(META_CACHE(sbi), index, + F2FS_CACHE_LOCK_CREATE); + if (IS_ERR(entry)) { cond_resched(); goto repeat; } - f2fs_folio_wait_writeback(folio, META, true, true); - if (!folio_test_uptodate(folio)) - folio_mark_uptodate(folio); - return folio; + f2fs_cache_wait_writeback(entry); + if (!f2fs_cache_test_uptodate(entry)) + f2fs_cache_set_uptodate(entry); + return entry; } -static struct folio *__get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index, - bool is_meta) +static struct f2fs_cached_block *__get_meta_cache(struct f2fs_sb_info *sbi, + pgoff_t index, bool is_meta) { - struct address_space *mapping = META_MAPPING(sbi); - struct folio *folio; + struct f2fs_cached_block *entry; struct f2fs_io_info fio = { .sbi = sbi, .type = META, @@ -265,70 +266,75 @@ static struct folio *__get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index, .new_blkaddr = index, .encrypted_page = NULL, .is_por = !is_meta ? 1 : 0, + .is_cache = 1, }; int err; if (unlikely(!is_meta)) fio.op_flags &= ~REQ_META; repeat: - folio = f2fs_grab_cache_folio(mapping, index, false); - if (IS_ERR(folio)) { + entry = f2fs_grab_cache(META_CACHE(sbi), index, + F2FS_CACHE_LOCK_CREATE); + if (IS_ERR(entry)) { cond_resched(); goto repeat; } - if (folio_test_uptodate(folio)) + if (f2fs_cache_test_uptodate(entry)) goto out; - fio.folio = folio; + fio.cache_entry = entry; - err = f2fs_submit_page_bio(&fio); + err = f2fs_submit_cache_read(&fio); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); return ERR_PTR(err); } f2fs_update_iostat(sbi, NULL, FS_META_READ_IO, F2FS_BLKSIZE); - folio_lock(folio); - if (unlikely(!is_meta_folio(folio))) { - f2fs_folio_put(folio, true); + f2fs_lock_cache(entry); + if (unlikely(!f2fs_is_meta_cache(entry))) { + f2fs_put_cache(entry, true); goto repeat; } - if (unlikely(!folio_test_uptodate(folio))) { - f2fs_handle_page_eio(sbi, folio, META); - f2fs_folio_put(folio, true); + if (unlikely(!f2fs_cache_test_uptodate(entry))) { + f2fs_handle_page_eio(sbi, entry->index, META); + f2fs_put_cache(entry, true); return ERR_PTR(-EIO); } out: - return folio; + return entry; } -struct folio *f2fs_get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index) +struct f2fs_cached_block *f2fs_get_meta_cache(struct f2fs_sb_info *sbi, + pgoff_t index) { - return __get_meta_folio(sbi, index, true); + return __get_meta_cache(sbi, index, true); } -struct folio *f2fs_get_meta_folio_retry(struct f2fs_sb_info *sbi, pgoff_t index) +struct f2fs_cached_block *f2fs_get_meta_cache_retry(struct f2fs_sb_info *sbi, + pgoff_t index) { - struct folio *folio; + struct f2fs_cached_block *entry; int count = 0; retry: - folio = __get_meta_folio(sbi, index, true); - if (IS_ERR(folio)) { - if (PTR_ERR(folio) == -EIO && + entry = __get_meta_cache(sbi, index, true); + if (IS_ERR(entry)) { + if (PTR_ERR(entry) == -EIO && ++count <= DEFAULT_RETRY_IO_COUNT) goto retry; f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_META_PAGE); } - return folio; + return entry; } /* for POR only */ -struct folio *f2fs_get_tmp_folio(struct f2fs_sb_info *sbi, pgoff_t index) +struct f2fs_cached_block *f2fs_get_tmp_cache(struct f2fs_sb_info *sbi, + pgoff_t index) { - return __get_meta_folio(sbi, index, false); + return __get_meta_cache(sbi, index, false); } static bool __is_bitmap_valid(struct f2fs_sb_info *sbi, block_t blkaddr, @@ -446,9 +452,10 @@ bool f2fs_is_valid_blkaddr_raw(struct f2fs_sb_info *sbi, /* * Readahead CP/NAT/SIT/SSA/POR pages */ -int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, - int type, bool sync) +int f2fs_ra_meta_caches(struct f2fs_sb_info *sbi, block_t start, + int nrpages, int type, bool sync) { + struct f2fs_cached_block_list *cache = META_CACHE(sbi); block_t blkno = start; struct f2fs_io_info fio = { .sbi = sbi, @@ -458,6 +465,7 @@ int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, .encrypted_page = NULL, .in_list = 0, .is_por = (type == META_POR) ? 1 : 0, + .is_cache = 1, }; struct blk_plug plug; int err; @@ -467,7 +475,7 @@ int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, blk_start_plug(&plug); for (; nrpages-- > 0; blkno++) { - struct folio *folio; + struct f2fs_cached_block *entry; if (!f2fs_is_valid_blkaddr(sbi, blkno, type)) goto out; @@ -494,62 +502,58 @@ int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, fio.new_blkaddr = blkno; break; default: - BUG(); + f2fs_bug_on(sbi, 1); } - folio = f2fs_grab_cache_folio(META_MAPPING(sbi), - fio.new_blkaddr, false); - if (IS_ERR(folio)) + entry = f2fs_grab_cache(cache, fio.new_blkaddr, + F2FS_CACHE_LOCK_CREATE); + if (IS_ERR(entry)) continue; - if (folio_test_uptodate(folio)) { - f2fs_folio_put(folio, true); + if (f2fs_cache_test_uptodate(entry)) { + f2fs_put_cache(entry, true); continue; } - fio.folio = folio; - err = f2fs_submit_page_bio(&fio); - f2fs_folio_put(folio, err ? true : false); + fio.cache_entry = entry; + err = f2fs_submit_cache_read(&fio); + f2fs_put_cache(entry, err ? true : false); if (!err) f2fs_update_iostat(sbi, NULL, FS_META_READ_IO, - F2FS_BLKSIZE); + sbi->blocksize); } out: blk_finish_plug(&plug); return blkno - start; } -void f2fs_ra_meta_pages_cond(struct f2fs_sb_info *sbi, pgoff_t index, - unsigned int ra_blocks) +void f2fs_ra_meta_caches_cond(struct f2fs_sb_info *sbi, pgoff_t index, + unsigned int ra_blocks) { - struct folio *folio; + struct f2fs_cached_block *entry; bool readahead = false; if (ra_blocks == RECOVERY_MIN_RA_BLOCKS) return; - folio = filemap_get_folio(META_MAPPING(sbi), index); - if (IS_ERR(folio) || !folio_test_uptodate(folio)) + entry = f2fs_find_cache(META_CACHE(sbi), index); + if (IS_ERR(entry) || !f2fs_cache_test_uptodate(entry)) readahead = true; - f2fs_folio_put(folio, false); + f2fs_put_cache(entry, false); if (readahead) - f2fs_ra_meta_pages(sbi, index, ra_blocks, META_POR, true); + f2fs_ra_meta_caches(sbi, index, ra_blocks, META_POR, true); } -static bool __f2fs_write_meta_folio(struct folio *folio, - struct writeback_control *wbc, +static bool __f2fs_write_meta_cache(struct f2fs_cached_block *entry, enum iostat_type io_type) { - struct f2fs_sb_info *sbi = F2FS_F_SB(folio); - - trace_f2fs_writepage(folio, META); + struct f2fs_sb_info *sbi = entry->cache->sbi; if (unlikely(f2fs_cp_error(sbi))) { if (is_sbi_flag_set(sbi, SBI_IS_CLOSE)) { - folio_clear_uptodate(folio); - dec_page_count(sbi, F2FS_DIRTY_META); - folio_unlock(folio); + f2fs_force_clear_cache_dirty(entry); + f2fs_unlock_cache(entry); return true; } goto redirty_out; @@ -557,10 +561,10 @@ static bool __f2fs_write_meta_folio(struct folio *folio, if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING))) goto redirty_out; - f2fs_do_write_meta_page(sbi, folio, io_type); + f2fs_do_write_meta_cache(sbi, entry, io_type); dec_page_count(sbi, F2FS_DIRTY_META); - folio_unlock(folio); + f2fs_unlock_cache(entry); if (unlikely(f2fs_cp_error(sbi))) f2fs_submit_merged_write(sbi, META); @@ -568,101 +572,89 @@ static bool __f2fs_write_meta_folio(struct folio *folio, return true; redirty_out: - folio_redirty_for_writepage(wbc, folio); + f2fs_cache_set_dirty(entry); return false; } -static int f2fs_write_meta_pages(struct address_space *mapping, - struct writeback_control *wbc) +void f2fs_write_meta_caches(struct f2fs_sb_info *sbi) { - struct f2fs_sb_info *sbi = F2FS_M_SB(mapping); struct f2fs_lock_context lc; - long diff, written; + long nr_to_write = LONG_MAX; if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING))) - goto skip_write; + return; - /* collect a number of dirty meta pages and write together */ - if (wbc->sync_mode != WB_SYNC_ALL && - get_pages(sbi, F2FS_DIRTY_META) < - nr_pages_to_skip(sbi, META)) - goto skip_write; + /* collect a number of dirty meta caches and write together */ + if (get_pages(sbi, F2FS_DIRTY_META) < + nr_pages_to_skip(sbi, META)) + return; - /* if locked failed, cp will flush dirty pages instead */ + /* if locked failed, cp will flush dirty caches instead */ if (!f2fs_down_write_trylock_trace(&sbi->cp_global_sem, &lc)) - goto skip_write; + return; - trace_f2fs_writepages(mapping->host, wbc, META); - diff = nr_pages_to_write(sbi, META, wbc); - written = f2fs_sync_meta_pages(sbi, wbc->nr_to_write, FS_META_IO); + nr_to_write = adjust_flush_cache_number(sbi, META); + f2fs_sync_meta_caches(sbi, nr_to_write, false, FS_META_IO); f2fs_up_write_trace(&sbi->cp_global_sem, &lc); - wbc->nr_to_write = max((long)0, wbc->nr_to_write - written - diff); - return 0; - -skip_write: - wbc->pages_skipped += get_pages(sbi, F2FS_DIRTY_META); - trace_f2fs_writepages(mapping->host, wbc, META); - return 0; } -long f2fs_sync_meta_pages(struct f2fs_sb_info *sbi, long nr_to_write, - enum iostat_type io_type) +long f2fs_sync_meta_caches(struct f2fs_sb_info *sbi, long nr_to_write, + bool sync, enum iostat_type io_type) { - struct address_space *mapping = META_MAPPING(sbi); pgoff_t index = 0, prev = ULONG_MAX; - struct folio_batch fbatch; + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; long nwritten = 0; - int nr_folios; - struct writeback_control wbc = {}; + int nr; struct blk_plug plug; - - folio_batch_init(&fbatch); + bool background = nr_to_write != LONG_MAX; blk_start_plug(&plug); - while ((nr_folios = filemap_get_folios_tag(mapping, &index, - (pgoff_t)-1, - PAGECACHE_TAG_DIRTY, &fbatch))) { + while ((nr = f2fs_cache_gang_lookup_tag(META_CACHE(sbi), entries, + &index, F2FS_ONSTACK_CACHES, F2FS_CACHE_TAG_DIRTY))) { int i; - for (i = 0; i < nr_folios; i++) { - struct folio *folio = fbatch.folios[i]; + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; + + if (background && unlikely(freezing(current))) { + f2fs_cache_gang_release(entries, nr); + goto stop; + } - if (nr_to_write != LONG_MAX && i != 0 && - folio->index != prev + - folio_nr_pages(fbatch.folios[i-1])) { - folio_batch_release(&fbatch); + if (background && i != 0 && + entry->index != prev + 1) { + f2fs_cache_gang_release(entries, nr); goto stop; } - folio_lock(folio); + f2fs_lock_cache(entry); - if (unlikely(!is_meta_folio(folio))) { + if (unlikely(!f2fs_is_meta_cache(entry))) { continue_unlock: - folio_unlock(folio); + f2fs_unlock_cache(entry); continue; } - if (!folio_test_dirty(folio)) { + if (!f2fs_cache_test_dirty(entry)) { /* someone wrote it for us */ goto continue_unlock; } - f2fs_folio_wait_writeback(folio, META, true, true); + f2fs_cache_wait_writeback(entry); - if (!folio_clear_dirty_for_io(folio)) + if (!f2fs_clear_cache_dirty(entry)) goto continue_unlock; - if (!__f2fs_write_meta_folio(folio, &wbc, - io_type)) { - folio_unlock(folio); + if (!__f2fs_write_meta_cache(entry, io_type)) { + f2fs_unlock_cache(entry); break; } - nwritten += folio_nr_pages(folio); - prev = folio->index; + nwritten += sbi->blocksize; + prev = entry->index; if (unlikely(nwritten >= nr_to_write)) break; } - folio_batch_release(&fbatch); + f2fs_cache_gang_release(entries, nr); cond_resched(); } stop: @@ -674,29 +666,6 @@ long f2fs_sync_meta_pages(struct f2fs_sb_info *sbi, long nr_to_write, return nwritten; } -static bool f2fs_dirty_meta_folio(struct address_space *mapping, - struct folio *folio) -{ - trace_f2fs_set_page_dirty(folio, META); - - if (!folio_test_uptodate(folio)) - folio_mark_uptodate(folio); - if (filemap_dirty_folio(mapping, folio)) { - inc_page_count(F2FS_M_SB(mapping), F2FS_DIRTY_META); - folio_set_f2fs_reference(folio); - return true; - } - return false; -} - -const struct address_space_operations f2fs_meta_aops = { - .writepages = f2fs_write_meta_pages, - .dirty_folio = f2fs_dirty_meta_folio, - .invalidate_folio = f2fs_invalidate_folio, - .release_folio = f2fs_release_folio, - .migrate_folio = filemap_migrate_folio, -}; - static void __add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, unsigned int devidx, int type) { @@ -1036,20 +1005,20 @@ int f2fs_recover_orphan_inodes(struct f2fs_sb_info *sbi) start_blk = __start_cp_addr(sbi) + 1 + __cp_payload(sbi); orphan_blocks = __start_sum_addr(sbi) - 1 - __cp_payload(sbi); - f2fs_ra_meta_pages(sbi, start_blk, orphan_blocks, META_CP, true); + f2fs_ra_meta_caches(sbi, start_blk, orphan_blocks, META_CP, true); for (i = 0; i < orphan_blocks; i++) { - struct folio *folio; + struct f2fs_cached_block *entry; struct f2fs_orphan_block *orphan_blk; unsigned int entry_count; - folio = f2fs_get_meta_folio(sbi, start_blk + i); - if (IS_ERR(folio)) { - err = PTR_ERR(folio); + entry = f2fs_get_meta_cache(sbi, start_blk + i); + if (IS_ERR(entry)) { + err = PTR_ERR(entry); goto out; } - orphan_blk = folio_address(folio); + orphan_blk = cache_address(entry); entry_count = le32_to_cpu(orphan_blk->entry_count); if (entry_count > F2FS_ORPHANS_PER_BLOCK) { f2fs_err(sbi, "invalid orphan inode entry count %u", @@ -1057,7 +1026,7 @@ int f2fs_recover_orphan_inodes(struct f2fs_sb_info *sbi) set_sbi_flag(sbi, SBI_NEED_FSCK); f2fs_handle_error(sbi, ERROR_INCONSISTENT_ORPHAN); err = -EFSCORRUPTED; - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); goto out; } @@ -1066,11 +1035,11 @@ int f2fs_recover_orphan_inodes(struct f2fs_sb_info *sbi) err = recover_orphan_inode(sbi, ino); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); goto out; } } - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); } /* clear Orphan Flag */ clear_ckpt_flags(sbi, CP_ORPHAN_PRESENT_FLAG); @@ -1087,9 +1056,9 @@ static void write_orphan_inodes(struct f2fs_sb_info *sbi, block_t start_blk) unsigned int nentries = 0; unsigned short index = 1; unsigned short orphan_blocks; - struct folio *folio = NULL; struct ino_entry *orphan = NULL; struct inode_management *im = &sbi->im[ORPHAN_INO]; + struct f2fs_cached_block *entry = NULL; orphan_blocks = GET_ORPHAN_BLOCKS(im->ino_num); @@ -1102,9 +1071,9 @@ static void write_orphan_inodes(struct f2fs_sb_info *sbi, block_t start_blk) /* loop for each orphan inode entry and write them in journal block */ list_for_each_entry(orphan, head, list) { - if (!folio) { - folio = f2fs_grab_meta_folio(sbi, start_blk++); - orphan_blk = folio_address(folio); + if (!entry) { + entry = f2fs_grab_meta_cache(sbi, start_blk++); + orphan_blk = (struct f2fs_orphan_block *)cache_address(entry); memset(orphan_blk, 0, sizeof(*orphan_blk)); } @@ -1119,20 +1088,20 @@ static void write_orphan_inodes(struct f2fs_sb_info *sbi, block_t start_blk) orphan_blk->blk_addr = cpu_to_le16(index); orphan_blk->blk_count = cpu_to_le16(orphan_blocks); orphan_blk->entry_count = cpu_to_le32(nentries); - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); + f2fs_mark_cache_dirty(entry); + f2fs_put_cache(entry, true); index++; nentries = 0; - folio = NULL; + entry = NULL; } } - if (folio) { + if (entry) { orphan_blk->blk_addr = cpu_to_le16(index); orphan_blk->blk_count = cpu_to_le16(orphan_blocks); orphan_blk->entry_count = cpu_to_le32(nentries); - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); + f2fs_mark_cache_dirty(entry); + f2fs_put_cache(entry, true); } } @@ -1151,29 +1120,29 @@ static __u32 f2fs_checkpoint_chksum(struct f2fs_checkpoint *ckpt) } static int get_checkpoint_version(struct f2fs_sb_info *sbi, block_t cp_addr, - struct f2fs_checkpoint **cp_block, struct folio **cp_folio, + struct f2fs_checkpoint **cp_block, struct f2fs_cached_block **cp_entry, unsigned long long *version) { size_t crc_offset = 0; __u32 crc; - *cp_folio = f2fs_get_meta_folio(sbi, cp_addr); - if (IS_ERR(*cp_folio)) - return PTR_ERR(*cp_folio); + *cp_entry = f2fs_get_meta_cache(sbi, cp_addr); + if (IS_ERR(*cp_entry)) + return PTR_ERR(*cp_entry); - *cp_block = folio_address(*cp_folio); + *cp_block = (struct f2fs_checkpoint *)cache_address(*cp_entry); crc_offset = le32_to_cpu((*cp_block)->checksum_offset); if (crc_offset < CP_MIN_CHKSUM_OFFSET || crc_offset > CP_CHKSUM_OFFSET) { - f2fs_folio_put(*cp_folio, true); + f2fs_put_cache(*cp_entry, true); f2fs_warn(sbi, "invalid crc_offset: %zu", crc_offset); return -EINVAL; } crc = f2fs_checkpoint_chksum(*cp_block); if (crc != cur_cp_crc(*cp_block)) { - f2fs_folio_put(*cp_folio, true); + f2fs_put_cache(*cp_entry, true); f2fs_warn(sbi, "invalid crc value"); return -EINVAL; } @@ -1182,17 +1151,17 @@ static int get_checkpoint_version(struct f2fs_sb_info *sbi, block_t cp_addr, return 0; } -static struct folio *validate_checkpoint(struct f2fs_sb_info *sbi, +static struct f2fs_cached_block *validate_checkpoint(struct f2fs_sb_info *sbi, block_t cp_addr, unsigned long long *version) { - struct folio *cp_folio_1 = NULL, *cp_folio_2 = NULL; + struct f2fs_cached_block *cp_entry_1 = NULL, *cp_entry_2 = NULL; struct f2fs_checkpoint *cp_block = NULL; unsigned long long cur_version = 0, pre_version = 0; unsigned int cp_blocks; int err; err = get_checkpoint_version(sbi, cp_addr, &cp_block, - &cp_folio_1, version); + &cp_entry_1, version); if (err) return NULL; @@ -1207,19 +1176,20 @@ static struct folio *validate_checkpoint(struct f2fs_sb_info *sbi, cp_addr += cp_blocks - 1; err = get_checkpoint_version(sbi, cp_addr, &cp_block, - &cp_folio_2, version); + &cp_entry_2, version); if (err) goto invalid_cp; cur_version = *version; if (cur_version == pre_version) { *version = cur_version; - f2fs_folio_put(cp_folio_2, true); - return cp_folio_1; + f2fs_put_cache(cp_entry_2, true); + return cp_entry_1; } - f2fs_folio_put(cp_folio_2, true); + f2fs_put_cache(cp_entry_2, true); invalid_cp: - f2fs_folio_put(cp_folio_1, true); + if (cp_entry_1) + f2fs_put_cache(cp_entry_1, true); return NULL; } @@ -1227,7 +1197,7 @@ int f2fs_get_valid_checkpoint(struct f2fs_sb_info *sbi) { struct f2fs_checkpoint *cp_block; struct f2fs_super_block *fsb = sbi->raw_super; - struct folio *cp1, *cp2, *cur_folio; + struct f2fs_cached_block *cp1 = NULL, *cp2 = NULL, *cur_entry; unsigned long blk_size = sbi->blocksize; unsigned long long cp1_version = 0, cp2_version = 0; unsigned long long cp_start_blk_no; @@ -1254,22 +1224,22 @@ int f2fs_get_valid_checkpoint(struct f2fs_sb_info *sbi) if (cp1 && cp2) { if (ver_after(cp2_version, cp1_version)) - cur_folio = cp2; + cur_entry = cp2; else - cur_folio = cp1; + cur_entry = cp1; } else if (cp1) { - cur_folio = cp1; + cur_entry = cp1; } else if (cp2) { - cur_folio = cp2; + cur_entry = cp2; } else { err = -EFSCORRUPTED; goto fail_no_cp; } - cp_block = folio_address(cur_folio); + cp_block = (struct f2fs_checkpoint *)cache_address(cur_entry); memcpy(sbi->ckpt, cp_block, blk_size); - if (cur_folio == cp1) + if (cur_entry == cp1) sbi->cur_cp_pack = 1; else sbi->cur_cp_pack = 2; @@ -1284,30 +1254,35 @@ int f2fs_get_valid_checkpoint(struct f2fs_sb_info *sbi) goto done; cp_blk_no = le32_to_cpu(fsb->cp_blkaddr); - if (cur_folio == cp2) + if (cur_entry == cp2) cp_blk_no += BIT(le32_to_cpu(fsb->log_blocks_per_seg)); for (i = 1; i < cp_blks; i++) { + struct f2fs_cached_block *cur_entry_payload; void *sit_bitmap_ptr; unsigned char *ckpt = (unsigned char *)sbi->ckpt; - cur_folio = f2fs_get_meta_folio(sbi, cp_blk_no + i); - if (IS_ERR(cur_folio)) { - err = PTR_ERR(cur_folio); + cur_entry_payload = f2fs_get_meta_cache(sbi, cp_blk_no + i); + if (IS_ERR(cur_entry_payload)) { + err = PTR_ERR(cur_entry_payload); goto free_fail_no_cp; } - sit_bitmap_ptr = folio_address(cur_folio); + sit_bitmap_ptr = cache_address(cur_entry_payload); memcpy(ckpt + i * blk_size, sit_bitmap_ptr, blk_size); - f2fs_folio_put(cur_folio, true); + f2fs_put_cache(cur_entry_payload, true); } done: - f2fs_folio_put(cp1, true); - f2fs_folio_put(cp2, true); + if (cp1) + f2fs_put_cache(cp1, true); + if (cp2) + f2fs_put_cache(cp2, true); return 0; free_fail_no_cp: - f2fs_folio_put(cp1, true); - f2fs_folio_put(cp2, true); + if (cp1) + f2fs_put_cache(cp1, true); + if (cp2) + f2fs_put_cache(cp2, true); fail_no_cp: kvfree(sbi->ckpt); return err; @@ -1621,7 +1596,7 @@ void f2fs_wait_on_all_pages(struct f2fs_sb_info *sbi, int type) break; if (type == F2FS_DIRTY_META) - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_CP_META_IO); + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_META_IO); else if (type == F2FS_WB_CP_DATA) f2fs_submit_merged_write(sbi, DATA); @@ -1700,31 +1675,24 @@ static void update_ckpt_flags(struct f2fs_sb_info *sbi, struct cp_control *cpc) static void commit_checkpoint(struct f2fs_sb_info *sbi, void *src, block_t blk_addr) { - struct writeback_control wbc = {}; - - /* - * filemap_get_folios_tag and folio_lock again will take - * some extra time. Therefore, f2fs_update_meta_pages and - * f2fs_sync_meta_pages are combined in this function. - */ - struct folio *folio = f2fs_grab_meta_folio(sbi, blk_addr); + struct f2fs_cached_block *entry = f2fs_grab_meta_cache(sbi, blk_addr); - memcpy(folio_address(folio), src, PAGE_SIZE); + memcpy(cache_address(entry), src, F2FS_BLKSIZE); - folio_mark_dirty(folio); - if (unlikely(!folio_clear_dirty_for_io(folio))) + f2fs_mark_cache_dirty(entry); + if (unlikely(!f2fs_clear_cache_dirty(entry))) f2fs_bug_on(sbi, 1); /* writeout cp pack 2 page */ - if (unlikely(!__f2fs_write_meta_folio(folio, &wbc, FS_CP_META_IO))) { + if (unlikely(!__f2fs_write_meta_cache(entry, FS_CP_META_IO))) { if (f2fs_cp_error(sbi)) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); return; } f2fs_bug_on(sbi, true); } - f2fs_folio_put(folio, false); + f2fs_put_cache(entry, false); /* submit checkpoint (with barrier if NOBARRIER is not set) */ f2fs_submit_merged_write(sbi, META_FLUSH); @@ -1793,7 +1761,7 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc) int err; /* Flush all the NAT/SIT pages */ - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_CP_META_IO); + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_CP_META_IO); stat_cp_time(cpc, CP_TIME_SYNC_META); @@ -1892,7 +1860,7 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc) } /* Here, we have one bio having CP pack except cp pack 2 page */ - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_CP_META_IO); + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_CP_META_IO); stat_cp_time(cpc, CP_TIME_SYNC_CP_META); /* Wait for all dirty meta pages to be submitted for IO */ @@ -1919,10 +1887,10 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc) * used for migration of encrypted, verity or compressed inode's blocks. */ if (f2fs_sb_has_encrypt(sbi) || f2fs_sb_has_verity(sbi) || - f2fs_sb_has_compression(sbi)) - f2fs_bug_on(sbi, - invalidate_inode_pages2_range(META_MAPPING(sbi), - MAIN_BLKADDR(sbi), MAX_BLKADDR(sbi) - 1)); + f2fs_sb_has_compression(sbi)) { + f2fs_truncate_meta_caches(sbi, MAIN_BLKADDR(sbi), + MAX_BLKADDR(sbi) - MAIN_BLKADDR(sbi)); + } f2fs_release_ino_entry(sbi, false); diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c index 91855d91bbdd..bb749f6257a1 100644 --- a/fs/f2fs/compress.c +++ b/fs/f2fs/compress.c @@ -1150,7 +1150,7 @@ static int prepare_compress_overwrite(struct compress_ctx *cc, f2fs_compress_ctx_add_page(cc, folio); if (!folio_test_uptodate(folio)) { - f2fs_handle_page_eio(sbi, folio, DATA); + f2fs_handle_page_eio(sbi, folio->index, DATA); release_and_retry: f2fs_put_rpages(cc); f2fs_unlock_rpages(cc, i + 1); @@ -1359,7 +1359,7 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc, fio.old_blkaddr = data_blkaddr(dn.inode, dn.node_folio, dn.ofs_in_node + i + 1); - /* wait for GCed page writeback via META_MAPPING */ + /* wait for GCed page writeback via generic cache */ f2fs_wait_on_block_writeback(inode, fio.old_blkaddr); if (fio.encrypted) { diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index 110282bb8dcd..92c3293f0a1e 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -65,8 +65,7 @@ bool f2fs_is_cp_guaranteed(const struct folio *folio) inode = mapping->host; - if (inode->i_ino == F2FS_META_INO(sbi) || - inode->i_ino == F2FS_NODE_INO(sbi) || + if (inode->i_ino == F2FS_NODE_INO(sbi) || S_ISDIR(inode->i_mode)) return true; @@ -84,9 +83,6 @@ static enum count_type __read_io_type(struct folio *folio) struct inode *inode = mapping->host; struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - if (inode->i_ino == F2FS_META_INO(sbi)) - return F2FS_RD_META; - if (inode->i_ino == F2FS_NODE_INO(sbi)) return F2FS_RD_NODE; } @@ -1452,7 +1448,7 @@ static void f2fs_submit_page_read(struct inode *inode, struct fsverity_info *vi, bio = f2fs_grab_read_bio(inode, vi, blkaddr, 1, op_flags, folio->index, for_write); - /* wait for GCed page writeback via META_MAPPING */ + /* wait for GCed page writeback via generic cache */ f2fs_wait_on_block_writeback(inode, blkaddr); if (!bio_add_folio(bio, folio, PAGE_SIZE, 0)) @@ -3110,7 +3106,7 @@ static void f2fs_readahead(struct readahead_control *rac) int f2fs_encrypt_one_page(struct f2fs_io_info *fio) { struct inode *inode = fio_inode(fio); - struct folio *mfolio; + struct f2fs_cached_block *entry; struct page *page; if (!f2fs_encrypted_file(inode)) @@ -3126,12 +3122,13 @@ int f2fs_encrypt_one_page(struct f2fs_io_info *fio) if (IS_ERR(fio->encrypted_page)) return PTR_ERR(fio->encrypted_page); - mfolio = filemap_lock_folio(META_MAPPING(fio->sbi), fio->old_blkaddr); - if (!IS_ERR(mfolio)) { - if (folio_test_uptodate(mfolio)) - memcpy(folio_address(mfolio), - page_address(fio->encrypted_page), PAGE_SIZE); - f2fs_folio_put(mfolio, true); + entry = f2fs_find_cache(META_CACHE(fio->sbi), fio->old_blkaddr); + if (!IS_ERR(entry)) { + f2fs_lock_cache(entry); + if (f2fs_cache_test_uptodate(entry)) + memcpy(cache_address(entry), page_address(fio->encrypted_page), + F2FS_BLKSIZE); + f2fs_put_cache(entry, true); } return 0; } @@ -3297,7 +3294,7 @@ int f2fs_do_write_data_page(struct f2fs_io_info *fio) goto out_writepage; } - /* wait for GCed page writeback via META_MAPPING */ + /* wait for GCed page writeback via generic cache */ if (fio->meta_gc) f2fs_wait_on_block_writeback(inode, fio->old_blkaddr); @@ -4380,9 +4377,7 @@ void f2fs_invalidate_folio(struct folio *folio, size_t offset, size_t length) return; if (folio_test_dirty(folio)) { - if (inode->i_ino == F2FS_META_INO(sbi)) { - dec_page_count(sbi, F2FS_DIRTY_META); - } else if (inode->i_ino == F2FS_NODE_INO(sbi)) { + if (inode->i_ino == F2FS_NODE_INO(sbi)) { dec_page_count(sbi, F2FS_DIRTY_NODES); } else { inode_dec_dirty_pages(inode); diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c index ff379aff4472..14059a50739c 100644 --- a/fs/f2fs/debug.c +++ b/fs/f2fs/debug.c @@ -224,8 +224,7 @@ static void update_general_status(struct f2fs_sb_info *sbi) si->dirty_count = dirty_segments(sbi); if (sbi->node_inode) si->node_pages = NODE_MAPPING(sbi)->nrpages; - if (sbi->meta_inode) - si->meta_pages = META_MAPPING(sbi)->nrpages; + si->meta_caches = META_CACHE(sbi)->num_entries; #ifdef CONFIG_F2FS_FS_COMPRESSION if (sbi->compress_inode) { si->compress_pages = COMPRESS_MAPPING(sbi)->nrpages; @@ -388,11 +387,8 @@ static void update_mem_info(struct f2fs_sb_info *sbi) si->page_mem += (unsigned long long)npages << PAGE_SHIFT; } - if (sbi->meta_inode) { - unsigned long npages = META_MAPPING(sbi)->nrpages; - - si->page_mem += (unsigned long long)npages << PAGE_SHIFT; - } + si->page_mem += (unsigned long long)META_CACHE(sbi)->num_entries << PAGE_SHIFT; + si->cache_mem += META_CACHE(sbi)->num_entries * sizeof(struct f2fs_cached_block); #ifdef CONFIG_F2FS_FS_COMPRESSION if (sbi->compress_inode) { unsigned long npages = COMPRESS_MAPPING(sbi)->nrpages; @@ -708,7 +704,7 @@ static int stat_show(struct seq_file *s, void *v) seq_printf(s, " - quota data: %4d in quota files:%4d\n", si->ndirty_qdata, si->nquota_files); seq_printf(s, " - meta: %4d in %4d\n", - si->ndirty_meta, si->meta_pages); + si->ndirty_meta, si->meta_caches); seq_printf(s, " - imeta: %4d\n", si->ndirty_imeta); seq_printf(s, " - fsync mark: %4lld\n", diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 6e20b3586f26..9a353dcd6658 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -1828,7 +1828,6 @@ struct f2fs_sb_info { struct f2fs_checkpoint *ckpt; /* raw checkpoint pointer */ int cur_cp_pack; /* remain current cp pack */ spinlock_t cp_lock; /* for flag in ckpt */ - struct inode *meta_inode; /* cache meta blocks */ struct f2fs_rwsem cp_global_sem; /* checkpoint procedure lock */ struct f2fs_rwsem cp_rwsem; /* blocking FS operations */ struct f2fs_rwsem node_write; /* locking node writes */ @@ -1873,7 +1872,6 @@ struct f2fs_sb_info { unsigned int blocksize; /* block size */ unsigned int root_ino_num; /* root inode number*/ unsigned int node_ino_num; /* node inode number*/ - unsigned int meta_ino_num; /* meta inode number*/ unsigned int log_blocks_per_seg; /* log2 blocks per segment */ unsigned int blocks_per_seg; /* blocks per segment */ unsigned int segs_per_sec; /* segments per section */ @@ -2319,9 +2317,9 @@ static inline struct dirty_seglist_info *DIRTY_I(struct f2fs_sb_info *sbi) return (struct dirty_seglist_info *)(SM_I(sbi)->dirty_info); } -static inline struct address_space *META_MAPPING(struct f2fs_sb_info *sbi) +static inline bool f2fs_is_meta_cache(struct f2fs_cached_block *entry) { - return sbi->meta_inode->i_mapping; + return entry->cache && entry->cache == META_CACHE(entry->cache->sbi); } static inline struct address_space *NODE_MAPPING(struct f2fs_sb_info *sbi) @@ -2329,11 +2327,6 @@ static inline struct address_space *NODE_MAPPING(struct f2fs_sb_info *sbi) return sbi->node_inode->i_mapping; } -static inline bool is_meta_folio(struct folio *folio) -{ - return folio->mapping == META_MAPPING(F2FS_F_SB(folio)); -} - static inline bool is_node_folio(struct folio *folio) { return folio->mapping == NODE_MAPPING(F2FS_F_SB(folio)); @@ -4057,9 +4050,9 @@ bool f2fs_alloc_nid(struct f2fs_sb_info *sbi, nid_t *nid); void f2fs_alloc_nid_done(struct f2fs_sb_info *sbi, nid_t nid); void f2fs_alloc_nid_failed(struct f2fs_sb_info *sbi, nid_t nid); int f2fs_try_to_free_nids(struct f2fs_sb_info *sbi, int nr_shrink); -int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio); -int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio); -int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct folio *folio); +int f2fs_recover_inline_xattr(struct inode *inode, struct f2fs_cached_block *entry); +int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry); +int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry); int f2fs_restore_node_summary(struct f2fs_sb_info *sbi, unsigned int segno, struct f2fs_summary_block *sum); int f2fs_flush_nat_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc); @@ -4109,11 +4102,13 @@ int f2fs_allocate_new_segments(struct f2fs_sb_info *sbi); int f2fs_trim_fs(struct f2fs_sb_info *sbi, struct fstrim_range *range); bool f2fs_exist_trim_candidates(struct f2fs_sb_info *sbi, struct cp_control *cpc); -struct folio *f2fs_get_sum_folio(struct f2fs_sb_info *sbi, unsigned int segno); +struct f2fs_cached_block *f2fs_get_sum_cache(struct f2fs_sb_info *sbi, + unsigned int segno); void f2fs_update_meta_page(struct f2fs_sb_info *sbi, void *src, block_t blk_addr); -void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct folio *folio, - enum iostat_type io_type); +void f2fs_do_write_meta_cache(struct f2fs_sb_info *sbi, + struct f2fs_cached_block *entry, + enum iostat_type io_type); void f2fs_do_write_node_page(unsigned int nid, struct f2fs_io_info *fio); void f2fs_outplace_write_data(struct dnode_of_data *dn, struct f2fs_io_info *fio); @@ -4136,8 +4131,6 @@ void f2fs_update_device_state(struct f2fs_sb_info *sbi, nid_t ino, block_t blkaddr, unsigned int blkcnt); void f2fs_folio_wait_writeback(struct folio *folio, enum page_type type, bool ordered, bool locked); -#define f2fs_wait_on_page_writeback(page, type, ordered, locked) \ - f2fs_folio_wait_writeback(page_folio(page), type, ordered, locked) void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr); void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr, block_t len); @@ -4203,20 +4196,21 @@ void f2fs_unlock_op(struct f2fs_sb_info *sbi, struct f2fs_lock_context *lc); void f2fs_stop_checkpoint(struct f2fs_sb_info *sbi, bool end_io, unsigned char reason); void f2fs_flush_ckpt_thread(struct f2fs_sb_info *sbi); -struct folio *f2fs_grab_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index); -struct folio *f2fs_get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index); -struct folio *f2fs_get_meta_folio_retry(struct f2fs_sb_info *sbi, pgoff_t index); -struct folio *f2fs_get_tmp_folio(struct f2fs_sb_info *sbi, pgoff_t index); +struct f2fs_cached_block *f2fs_grab_meta_cache(struct f2fs_sb_info *sbi, pgoff_t index); +struct f2fs_cached_block *f2fs_get_meta_cache(struct f2fs_sb_info *sbi, pgoff_t index); +struct f2fs_cached_block *f2fs_get_meta_cache_retry(struct f2fs_sb_info *sbi, pgoff_t index); +struct f2fs_cached_block *f2fs_get_tmp_cache(struct f2fs_sb_info *sbi, pgoff_t index); bool f2fs_is_valid_blkaddr(struct f2fs_sb_info *sbi, block_t blkaddr, int type); bool f2fs_is_valid_blkaddr_raw(struct f2fs_sb_info *sbi, block_t blkaddr, int type); -int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, +int f2fs_ra_meta_caches(struct f2fs_sb_info *sbi, block_t start, int nrpages, int type, bool sync); -void f2fs_ra_meta_pages_cond(struct f2fs_sb_info *sbi, pgoff_t index, +void f2fs_ra_meta_caches_cond(struct f2fs_sb_info *sbi, pgoff_t index, unsigned int ra_blocks); -long f2fs_sync_meta_pages(struct f2fs_sb_info *sbi, long nr_to_write, - enum iostat_type io_type); +void f2fs_write_meta_caches(struct f2fs_sb_info *sbi); +long f2fs_sync_meta_caches(struct f2fs_sb_info *sbi, long nr_to_write, + bool sync, enum iostat_type io_type); void f2fs_add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type); void f2fs_remove_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type); void f2fs_release_ino_entry(struct f2fs_sb_info *sbi, bool all); @@ -4403,7 +4397,7 @@ struct f2fs_stat_info { unsigned int bimodal, avg_vblocks; int util_free, util_valid, util_invalid; int rsvd_segs, overp_segs; - int dirty_count, node_pages, meta_pages, compress_pages; + int dirty_count, node_pages, meta_caches, compress_pages; int compress_page_hit; int prefree_count, free_segs, free_secs; int cp_call_count[MAX_CALL_TYPE], cp_count; @@ -4605,7 +4599,6 @@ extern const struct file_operations f2fs_file_operations; extern const struct inode_operations f2fs_file_inode_operations; extern const struct address_space_operations f2fs_dblock_aops; extern const struct address_space_operations f2fs_node_aops; -extern const struct address_space_operations f2fs_meta_aops; extern const struct inode_operations f2fs_dir_inode_operations; extern const struct inode_operations f2fs_symlink_inode_operations; extern const struct inode_operations f2fs_encrypted_symlink_inode_operations; @@ -4626,7 +4619,7 @@ int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio); int f2fs_convert_inline_inode(struct inode *inode); int f2fs_try_convert_inline_dir(struct inode *dir, struct dentry *dentry); int f2fs_write_inline_data(struct inode *inode, struct folio *folio); -int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio); +int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entry); struct f2fs_dir_entry *f2fs_find_in_inline_dir(struct inode *dir, const struct f2fs_filename *fname, struct folio **res_folio, bool use_hash); @@ -5186,10 +5179,8 @@ static inline void f2fs_schedule_timeout_killable(long timeout, bool io) } static inline void f2fs_handle_page_eio(struct f2fs_sb_info *sbi, - struct folio *folio, enum page_type type) + pgoff_t ofs, enum page_type type) { - pgoff_t ofs = folio->index; - if (unlikely(f2fs_cp_error(sbi))) return; @@ -5224,36 +5215,10 @@ static inline bool f2fs_is_readonly(struct f2fs_sb_info *sbi) return f2fs_sb_has_readonly(sbi) || f2fs_readonly(sbi->sb); } -static inline void f2fs_truncate_meta_inode_pages(struct f2fs_sb_info *sbi, - block_t blkaddr, unsigned int cnt) -{ - bool need_submit = false; - int i = 0; - - do { - struct folio *folio; - - folio = filemap_get_folio(META_MAPPING(sbi), blkaddr + i); - if (!IS_ERR(folio)) { - if (folio_test_writeback(folio)) - need_submit = true; - f2fs_folio_put(folio, false); - } - } while (++i < cnt && !need_submit); - - if (need_submit) - f2fs_submit_merged_write_cond(sbi, sbi->meta_inode, - NULL, 0, DATA); - - truncate_inode_pages_range(META_MAPPING(sbi), - F2FS_BLK_TO_BYTES((loff_t)blkaddr), - F2FS_BLK_END_BYTES((loff_t)(blkaddr + cnt - 1))); -} - static inline void f2fs_invalidate_internal_cache(struct f2fs_sb_info *sbi, block_t blkaddr, unsigned int len) { - f2fs_truncate_meta_inode_pages(sbi, blkaddr, len); + f2fs_truncate_meta_caches(sbi, blkaddr, len); f2fs_invalidate_compress_pages_range(sbi, blkaddr, len); } diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c index a54b3ab52f1a..92daa41dd96d 100644 --- a/fs/f2fs/file.c +++ b/fs/f2fs/file.c @@ -214,7 +214,7 @@ static vm_fault_t f2fs_vm_page_mkwrite(struct vm_fault *vmf) f2fs_folio_wait_writeback(folio, DATA, false, true); - /* wait for GCed page writeback via META_MAPPING */ + /* wait for GCed page writeback via generic cache */ f2fs_wait_on_block_writeback(inode, dn.data_blkaddr); /* @@ -2563,7 +2563,7 @@ int f2fs_do_shutdown(struct f2fs_sb_info *sbi, unsigned int flag, f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_SHUTDOWN); break; case F2FS_GOING_DOWN_METAFLUSH: - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_META_IO); + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_META_IO); f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_SHUTDOWN); break; case F2FS_GOING_DOWN_NEED_FSCK: diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c index c4da2f31805b..54327cb2e27e 100644 --- a/fs/f2fs/gc.c +++ b/fs/f2fs/gc.c @@ -1066,7 +1066,7 @@ static int gc_node_segment(struct f2fs_sb_info *sbi, continue; if (phase == 0) { - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), 1, + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nid), 1, META_NAT, true); continue; } @@ -1217,7 +1217,8 @@ static int ra_data_block(struct inode *inode, pgoff_t index) struct address_space *mapping = inode->i_mapping; struct inode *atomic_inode = NULL; struct dnode_of_data dn; - struct folio *folio, *efolio; + struct folio *folio; + struct f2fs_cached_block *entry; struct f2fs_io_info fio = { .sbi = sbi, .ino = inode->i_ino, @@ -1227,6 +1228,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index) .op_flags = 0, .encrypted_page = NULL, .in_list = 0, + .is_cache = 1, }; int err = 0; @@ -1285,22 +1287,21 @@ static int ra_data_block(struct inode *inode, pgoff_t index) f2fs_wait_on_block_writeback(inode, dn.data_blkaddr); - efolio = f2fs_filemap_get_folio(META_MAPPING(sbi), dn.data_blkaddr, - FGP_LOCK | FGP_CREAT, GFP_NOFS); - if (IS_ERR(efolio)) { - err = PTR_ERR(efolio); + entry = f2fs_grab_cache(META_CACHE(sbi), dn.data_blkaddr, + F2FS_CACHE_LOCK_CREATE); + if (IS_ERR(entry)) { + err = PTR_ERR(entry); goto put_folio; } - fio.encrypted_page = &efolio->page; - - if (folio_test_uptodate(efolio)) - goto put_encrypted_page; + if (f2fs_cache_test_uptodate(entry)) + goto put_cache; - err = f2fs_submit_page_bio(&fio); + fio.cache_entry = entry; + err = f2fs_submit_cache_read(&fio); if (err) - goto put_encrypted_page; - f2fs_put_page(fio.encrypted_page, false); + goto put_cache; + f2fs_put_cache(entry, false); f2fs_folio_put(folio, true); f2fs_update_iostat(sbi, inode, FS_DATA_READ_IO, F2FS_BLKSIZE); @@ -1309,8 +1310,8 @@ static int ra_data_block(struct inode *inode, pgoff_t index) if (atomic_inode) iput(atomic_inode); return 0; -put_encrypted_page: - f2fs_put_page(fio.encrypted_page, true); +put_cache: + f2fs_put_cache(entry, true); put_folio: f2fs_folio_put(folio, true); out_iput: @@ -1320,7 +1321,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index) } /* - * Move data block via META_MAPPING while keeping locked data page. + * Move data block via meta cache while keeping locked data page. * This can be used to move blocks, aka LBAs, directly on disk. */ static int move_data_block(struct inode *inode, block_t bidx, @@ -1337,11 +1338,13 @@ static int move_data_block(struct inode *inode, block_t bidx, .op_flags = 0, .encrypted_page = NULL, .in_list = 0, + .is_cache = 1, }; struct dnode_of_data dn; struct f2fs_summary sum; struct node_info ni; - struct folio *folio, *mfolio, *efolio; + struct folio *folio; + struct f2fs_cached_block *sentry, *tentry; block_t newaddr; int err = 0; bool lfs_mode = f2fs_lfs_mode(fio.sbi); @@ -1406,20 +1409,20 @@ static int move_data_block(struct inode *inode, block_t bidx, if (lfs_mode) f2fs_down_write(&fio.sbi->io_order_lock); - mfolio = f2fs_grab_cache_folio(META_MAPPING(fio.sbi), - fio.old_blkaddr, false); - if (IS_ERR(mfolio)) { - err = PTR_ERR(mfolio); + sentry = f2fs_grab_cache(META_CACHE(fio.sbi), fio.old_blkaddr, + F2FS_CACHE_LOCK_CREATE); + if (IS_ERR(sentry)) { + err = PTR_ERR(sentry); goto up_out; } - fio.encrypted_page = folio_file_page(mfolio, fio.old_blkaddr); + fio.cache_entry = sentry; - /* read source block in mfolio */ - if (!folio_test_uptodate(mfolio)) { - err = f2fs_submit_page_bio(&fio); + /* read source block in cache */ + if (!f2fs_cache_test_uptodate(sentry)) { + err = f2fs_submit_cache_read(&fio); if (err) { - f2fs_folio_put(mfolio, true); + f2fs_put_cache(sentry, true); goto up_out; } @@ -1428,11 +1431,11 @@ static int move_data_block(struct inode *inode, block_t bidx, f2fs_update_iostat(fio.sbi, NULL, FS_GDATA_READ_IO, F2FS_BLKSIZE); - folio_lock(mfolio); - if (unlikely(!is_meta_folio(mfolio) || - !folio_test_uptodate(mfolio))) { + f2fs_lock_cache(sentry); + if (unlikely(!f2fs_is_meta_cache(sentry) || + !f2fs_cache_test_uptodate(sentry))) { err = -EIO; - f2fs_folio_put(mfolio, true); + f2fs_put_cache(sentry, true); goto up_out; } } @@ -1443,46 +1446,45 @@ static int move_data_block(struct inode *inode, block_t bidx, err = f2fs_allocate_data_block(fio.sbi, NULL, fio.old_blkaddr, &newaddr, &sum, type, NULL); if (err) { - f2fs_folio_put(mfolio, true); + f2fs_put_cache(sentry, true); /* filesystem should shutdown, no need to recovery block */ goto up_out; } - efolio = f2fs_filemap_get_folio(META_MAPPING(fio.sbi), newaddr, - FGP_LOCK | FGP_CREAT, GFP_NOFS); - if (IS_ERR(efolio)) { - err = PTR_ERR(efolio); - f2fs_folio_put(mfolio, true); + tentry = f2fs_grab_cache(META_CACHE(fio.sbi), newaddr, + F2FS_CACHE_LOCK_CREATE); + if (IS_ERR(tentry)) { + err = PTR_ERR(tentry); + f2fs_put_cache(sentry, true); goto recover_block; } - fio.encrypted_page = &efolio->page; + fio.cache_entry = tentry; /* write target block */ - f2fs_wait_on_page_writeback(fio.encrypted_page, DATA, true, true); - memcpy(page_address(fio.encrypted_page), - folio_address(mfolio), PAGE_SIZE); - f2fs_folio_put(mfolio, true); + f2fs_cache_wait_writeback_cond(tentry, DATA); + memcpy(cache_address(tentry), cache_address(sentry), PAGE_SIZE); + f2fs_put_cache(sentry, true); f2fs_invalidate_internal_cache(fio.sbi, fio.old_blkaddr, 1); - set_page_dirty(fio.encrypted_page); - if (clear_page_dirty_for_io(fio.encrypted_page)) + f2fs_mark_cache_dirty(tentry); + if (f2fs_clear_cache_dirty(tentry)) dec_page_count(fio.sbi, F2FS_DIRTY_META); - set_page_writeback(fio.encrypted_page); + f2fs_start_cache_writeback(tentry); fio.op = REQ_OP_WRITE; fio.op_flags = REQ_SYNC; fio.new_blkaddr = newaddr; - f2fs_submit_page_write(&fio); + f2fs_submit_cache_write(&fio); f2fs_update_iostat(fio.sbi, NULL, FS_GC_DATA_IO, F2FS_BLKSIZE); f2fs_update_data_blkaddr(&dn, newaddr); set_inode_flag(inode, FI_APPEND_WRITE); - f2fs_put_page(fio.encrypted_page, true); + f2fs_put_cache(tentry, true); recover_block: if (err) f2fs_do_replace_block(fio.sbi, &sum, newaddr, fio.old_blkaddr, @@ -1614,7 +1616,7 @@ static int gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum, continue; if (phase == 0) { - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), 1, + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nid), 1, META_NAT, true); continue; } @@ -1818,28 +1820,31 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, sum_blk_cnt = DIV_ROUND_UP(end_segno - segno, sbi->sums_per_block); /* readahead multi ssa blocks those have contiguous address */ if (__is_large_section(sbi)) - f2fs_ra_meta_pages(sbi, GET_SUM_BLOCK(sbi, segno), + f2fs_ra_meta_caches(sbi, GET_SUM_BLOCK(sbi, segno), sum_blk_cnt, META_SSA, true); /* reference all summary page */ while (segno < end_segno) { - struct folio *sum_folio = f2fs_get_sum_folio(sbi, segno); + struct f2fs_cached_block *sum_entry = + f2fs_get_sum_cache(sbi, segno); segno += sbi->sums_per_block; - if (IS_ERR(sum_folio)) { - int err = PTR_ERR(sum_folio); + if (IS_ERR(sum_entry)) { + int err = PTR_ERR(sum_entry); end_segno = segno - sbi->sums_per_block; segno = rounddown(start_segno, sbi->sums_per_block); while (segno < end_segno) { - sum_folio = filemap_get_folio(META_MAPPING(sbi), + sum_entry = f2fs_find_meta_cache(sbi, GET_SUM_BLOCK(sbi, segno)); - folio_put_refs(sum_folio, 2); + f2fs_put_cache(sum_entry, false); + f2fs_put_cache(sum_entry, false); segno += sbi->sums_per_block; } return err; } - folio_unlock(sum_folio); + f2fs_unlock_cache(sum_entry); + } blk_start_plug(&plug); @@ -1847,11 +1852,16 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, segno = start_segno; while (segno < end_segno) { unsigned int cur_segno; + unsigned int block_end_segno; /* find segment summary of victim */ - struct folio *sum_folio = filemap_get_folio(META_MAPPING(sbi), + struct f2fs_cached_block *sum_entry = + f2fs_find_meta_cache(sbi, GET_SUM_BLOCK(sbi, segno)); - unsigned int block_end_segno = rounddown(segno, sbi->sums_per_block) + + f2fs_bug_on(sbi, IS_ERR(sum_entry)); + + block_end_segno = rounddown(segno, sbi->sums_per_block) + sbi->sums_per_block; if (block_end_segno > end_segno) @@ -1864,8 +1874,8 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, goto next_block; } - if (!folio_test_uptodate(sum_folio) || - unlikely(f2fs_cp_error(sbi))) + if (!f2fs_cache_test_uptodate(sum_entry) || + unlikely(f2fs_cp_error(sbi))) goto next_block; for (cur_segno = segno; cur_segno < block_end_segno; @@ -1884,7 +1894,7 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, data_type = (type == SUM_TYPE_DATA) ? DATA : NODE; } - sum = SUM_BLK_PAGE_ADDR(sbi, sum_folio, cur_segno); + sum = SUM_BLK_ENTRY_ADDR(sbi, sum_entry, cur_segno); if (type != GET_SUM_TYPE(sum_footer(sbi, sum))) { f2fs_err(sbi, "Inconsistent segment (%u) type " "[%d, %d] in SIT and SSA", @@ -1926,12 +1936,14 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, cur_segno + 1 : NULL_SEGNO; if (unlikely(freezing(current))) { - folio_put_refs(sum_folio, 2); + f2fs_put_cache(sum_entry, false); + f2fs_put_cache(sum_entry, false); goto stop; } } next_block: - folio_put_refs(sum_folio, 2); + f2fs_put_cache(sum_entry, false); + f2fs_put_cache(sum_entry, false); segno = block_end_segno; } diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c index aec06fb4fd76..2156fb1fc57d 100644 --- a/fs/f2fs/inline.c +++ b/fs/f2fs/inline.c @@ -294,7 +294,7 @@ int f2fs_write_inline_data(struct inode *inode, struct folio *folio) return 0; } -int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio) +int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entry) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct f2fs_inode *ri = NULL; @@ -308,12 +308,13 @@ int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio) * x o -> remove data blocks, and then recover inline_data * x x -> recover data blocks */ - if (IS_INODE(nfolio)) - ri = F2FS_INODE(nfolio); + if (IS_INODE(cache_folio(entry))) + ri = &CACHED_NODE(entry)->i; if (f2fs_has_inline_data(inode) && ri && (ri->i_inline & F2FS_INLINE_DATA)) { struct folio *ifolio; + process_inline: ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); if (IS_ERR(ifolio)) @@ -321,7 +322,7 @@ int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio) f2fs_folio_wait_writeback(ifolio, NODE, true, true); - src_addr = inline_data_addr(inode, nfolio); + src_addr = inline_data_addr(inode, cache_folio(entry)); dst_addr = inline_data_addr(inode, ifolio); memcpy(dst_addr, src_addr, MAX_INLINE_DATA(inode)); diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c index bac1e360d966..c533da4d4d70 100644 --- a/fs/f2fs/inode.c +++ b/fs/f2fs/inode.c @@ -577,7 +577,7 @@ static int do_read_inode(struct inode *inode) static bool is_meta_ino(struct f2fs_sb_info *sbi, unsigned int ino) { - if (ino == F2FS_NODE_INO(sbi) || ino == F2FS_META_INO(sbi)) + if (ino == F2FS_NODE_INO(sbi)) return true; #ifdef CONFIG_F2FS_FS_COMPRESSION if (test_opt(sbi, COMPRESS_CACHE) && ino == F2FS_COMPRESS_INO(sbi)) @@ -624,9 +624,6 @@ struct inode *f2fs_iget(struct super_block *sb, unsigned long ino) if (ino == F2FS_NODE_INO(sbi)) { inode->i_mapping->a_ops = &f2fs_node_aops; mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS); - } else if (ino == F2FS_META_INO(sbi)) { - inode->i_mapping->a_ops = &f2fs_meta_aops; - mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS); } else if (ino == F2FS_COMPRESS_INO(sbi)) { #ifdef CONFIG_F2FS_FS_COMPRESSION inode->i_mapping->a_ops = &f2fs_compress_aops; @@ -824,8 +821,7 @@ int f2fs_write_inode(struct inode *inode, struct writeback_control *wbc) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - if (inode->i_ino == F2FS_NODE_INO(sbi) || - inode->i_ino == F2FS_META_INO(sbi)) + if (inode->i_ino == F2FS_NODE_INO(sbi)) return 0; /* @@ -919,7 +915,6 @@ static bool f2fs_pre_evict_inode(struct inode *inode) f2fs_invalidate_compress_pages(sbi, inode->i_ino); if (inode->i_ino == F2FS_NODE_INO(sbi) || - inode->i_ino == F2FS_META_INO(sbi) || inode->i_ino == F2FS_COMPRESS_INO(sbi)) return true; diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c index 46bea52e35c3..398c58fc6cad 100644 --- a/fs/f2fs/node.c +++ b/fs/f2fs/node.c @@ -140,38 +140,36 @@ static void clear_node_folio_dirty(struct folio *folio) folio_clear_uptodate(folio); } -static struct folio *get_current_nat_folio(struct f2fs_sb_info *sbi, nid_t nid) +static struct f2fs_cached_block *get_current_nat_cache(struct f2fs_sb_info *sbi, + nid_t nid) { - return f2fs_get_meta_folio_retry(sbi, current_nat_addr(sbi, nid)); + return f2fs_get_meta_cache_retry(sbi, current_nat_addr(sbi, nid)); } -static struct folio *get_next_nat_folio(struct f2fs_sb_info *sbi, nid_t nid) +static struct f2fs_cached_block *get_next_nat_cache(struct f2fs_sb_info *sbi, + nid_t nid) { - struct folio *src_folio; - struct folio *dst_folio; + struct f2fs_cached_block *src_entry; + struct f2fs_cached_block *dst_entry; pgoff_t dst_off; - void *src_addr; - void *dst_addr; struct f2fs_nm_info *nm_i = NM_I(sbi); dst_off = next_nat_addr(sbi, current_nat_addr(sbi, nid)); /* get current nat block page with lock */ - src_folio = get_current_nat_folio(sbi, nid); - if (IS_ERR(src_folio)) - return src_folio; - dst_folio = f2fs_grab_meta_folio(sbi, dst_off); - f2fs_bug_on(sbi, folio_test_dirty(src_folio)); - - src_addr = folio_address(src_folio); - dst_addr = folio_address(dst_folio); - memcpy(dst_addr, src_addr, PAGE_SIZE); - folio_mark_dirty(dst_folio); - f2fs_folio_put(src_folio, true); + src_entry = get_current_nat_cache(sbi, nid); + if (IS_ERR(src_entry)) + return src_entry; + dst_entry = f2fs_grab_meta_cache(sbi, dst_off); + f2fs_bug_on(sbi, f2fs_cache_test_dirty(src_entry)); + + memcpy(cache_address(dst_entry), cache_address(src_entry), sbi->blocksize); + f2fs_mark_cache_dirty(dst_entry); + f2fs_put_cache(src_entry, true); set_to_next_nat(nm_i, nid); - return dst_folio; + return dst_entry; } static struct nat_entry *__alloc_nat_entry(struct f2fs_sb_info *sbi, @@ -575,7 +573,7 @@ int f2fs_get_node_info(struct f2fs_sb_info *sbi, nid_t nid, struct f2fs_journal *journal = curseg->journal; nid_t start_nid = START_NID(nid); struct f2fs_nat_block *nat_blk; - struct folio *folio = NULL; + struct f2fs_cached_block *entry = NULL; struct f2fs_nat_entry ne; struct nat_entry *e; pgoff_t index; @@ -629,14 +627,14 @@ int f2fs_get_node_info(struct f2fs_sb_info *sbi, nid_t nid, index = current_nat_addr(sbi, nid); f2fs_up_read(&nm_i->nat_tree_lock); - folio = f2fs_get_meta_folio(sbi, index); - if (IS_ERR(folio)) - return PTR_ERR(folio); + entry = f2fs_get_meta_cache(sbi, index); + if (IS_ERR(entry)) + return PTR_ERR(entry); - nat_blk = folio_address(folio); + nat_blk = cache_address(entry); ne = nat_blk->entries[nid - start_nid]; node_info_from_raw_nat(ni, &ne); - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); sanity_check: if (__is_valid_data_blkaddr(ni->blk_addr) && !f2fs_is_valid_blkaddr(sbi, ni->blk_addr, @@ -1622,7 +1620,7 @@ static struct folio *__get_node_folio(struct f2fs_sb_info *sbi, pgoff_t nid, out_put_err: /* ENOENT comes from read_node_folio which is not an error. */ if (err != -ENOENT) - f2fs_handle_page_eio(sbi, folio, NODE); + f2fs_handle_page_eio(sbi, folio->index, NODE); f2fs_folio_put(folio, true); return ERR_PTR(err); } @@ -2598,6 +2596,7 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, bool sync, bool mount) { struct f2fs_nm_info *nm_i = NM_I(sbi); + struct f2fs_cached_block *entry = NULL; int i = 0, ret; nid_t nid = nm_i->next_scan_nid; @@ -2623,7 +2622,7 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, } /* readahead nat pages to be scanned */ - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), FREE_NID_PAGES, + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nid), FREE_NID_PAGES, META_NAT, true); f2fs_down_read(&nm_i->nat_tree_lock); @@ -2631,14 +2630,14 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, while (1) { if (!test_bit_le(NAT_BLOCK_OFFSET(nid), nm_i->nat_block_bitmap)) { - struct folio *folio = get_current_nat_folio(sbi, nid); + entry = get_current_nat_cache(sbi, nid); - if (IS_ERR(folio)) { - ret = PTR_ERR(folio); + if (IS_ERR(entry)) { + ret = PTR_ERR(entry); } else { - ret = scan_nat_page(sbi, folio_address(folio), + ret = scan_nat_page(sbi, cache_address(entry), nid); - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); } if (ret) { @@ -2671,7 +2670,7 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, f2fs_up_read(&nm_i->nat_tree_lock); - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nm_i->next_scan_nid), + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nm_i->next_scan_nid), nm_i->ra_nid_pages, META_NAT, false); return 0; @@ -2826,7 +2825,7 @@ int f2fs_try_to_free_nids(struct f2fs_sb_info *sbi, int nr_shrink) return nr - nr_shrink; } -int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) +int f2fs_recover_inline_xattr(struct inode *inode, struct f2fs_cached_block *entry) { void *src_addr, *dst_addr; size_t inline_size; @@ -2837,7 +2836,7 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) if (IS_ERR(ifolio)) return PTR_ERR(ifolio); - ri = F2FS_INODE(folio); + ri = &CACHED_NODE(entry)->i; if (ri->i_inline & F2FS_INLINE_XATTR) { if (!f2fs_has_inline_xattr(inode)) { set_inode_flag(inode, FI_INLINE_XATTR); @@ -2852,7 +2851,7 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) } dst_addr = inline_xattr_addr(inode, ifolio); - src_addr = inline_xattr_addr(inode, folio); + src_addr = inline_xattr_addr(inode, cache_folio(entry)); inline_size = inline_xattr_size(inode); f2fs_folio_wait_writeback(ifolio, NODE, true, true); @@ -2863,7 +2862,7 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) return 0; } -int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio) +int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); nid_t prev_xnid = F2FS_I(inode)->i_xattr_nid; @@ -2901,8 +2900,8 @@ int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio) f2fs_update_inode_page(inode); /* 3: update and set xattr node page dirty */ - if (folio) { - memcpy(F2FS_NODE(xfolio), F2FS_NODE(folio), + if (entry) { + memcpy(F2FS_NODE(xfolio), CACHED_NODE(entry), VALID_XATTR_BLOCK_SIZE); folio_mark_dirty(xfolio); } @@ -2911,10 +2910,10 @@ int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio) return 0; } -int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct folio *folio) +int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) { struct f2fs_inode *src, *dst; - nid_t ino = ino_of_node(folio); + nid_t ino = ino_of_node(cache_folio(entry)); struct node_info old_ni, new_ni; struct folio *ifolio; int err; @@ -2940,7 +2939,7 @@ int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct folio *folio) fill_node_footer(ifolio, ino, ino, 0, true); set_cold_node(ifolio, false); - src = F2FS_INODE(folio); + src = &CACHED_NODE(entry)->i; dst = F2FS_INODE(ifolio); memcpy(dst, src, offsetof(struct f2fs_inode, i_ext)); @@ -2999,24 +2998,24 @@ int f2fs_restore_node_summary(struct f2fs_sb_info *sbi, nrpages = bio_max_segs(last_offset - i); /* readahead node pages */ - f2fs_ra_meta_pages(sbi, addr, nrpages, META_POR, true); + f2fs_ra_meta_caches(sbi, addr, nrpages, META_POR, true); for (idx = addr; idx < addr + nrpages; idx++) { - struct folio *folio = f2fs_get_tmp_folio(sbi, idx); + struct f2fs_cached_block *entry = + f2fs_get_tmp_cache(sbi, idx); - if (IS_ERR(folio)) - return PTR_ERR(folio); + if (IS_ERR(entry)) + return PTR_ERR(entry); - rn = F2FS_NODE(folio); + rn = CACHED_NODE(entry); sum_entry->nid = rn->footer.nid; sum_entry->version = 0; sum_entry->ofs_in_node = 0; sum_entry++; - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); } - invalidate_mapping_pages(META_MAPPING(sbi), addr, - addr + nrpages); + f2fs_truncate_meta_caches(sbi, addr, nrpages); } return 0; } @@ -3126,7 +3125,7 @@ static int __flush_nat_entry_set(struct f2fs_sb_info *sbi, bool to_journal = true; struct f2fs_nat_block *nat_blk; struct nat_entry *ne, *cur; - struct folio *folio = NULL; + struct f2fs_cached_block *entry = NULL; /* * there are two steps to flush nat entries: @@ -3140,11 +3139,11 @@ static int __flush_nat_entry_set(struct f2fs_sb_info *sbi, if (to_journal) { down_write(&curseg->journal_rwsem); } else { - folio = get_next_nat_folio(sbi, start_nid); - if (IS_ERR(folio)) - return PTR_ERR(folio); + entry = get_next_nat_cache(sbi, start_nid); + if (IS_ERR(entry)) + return PTR_ERR(entry); - nat_blk = folio_address(folio); + nat_blk = cache_address(entry); f2fs_bug_on(sbi, !nat_blk); } @@ -3181,7 +3180,7 @@ static int __flush_nat_entry_set(struct f2fs_sb_info *sbi, up_write(&curseg->journal_rwsem); } else { __update_nat_bits(sbi, start_nid, nat_blk); - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); } /* Allow dirty nats by node block allocation in write_begin */ @@ -3252,7 +3251,7 @@ int f2fs_flush_nat_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) __has_cursum_space(sbi, journal, entry_count, NAT_JOURNAL)) continue; - f2fs_ra_meta_pages(sbi, set->set, 1, META_NAT, true); + f2fs_ra_meta_caches(sbi, set->set, 1, META_NAT, true); } /* flush dirty nats in nat entry set */ list_for_each_entry_safe(set, tmp, &sets, set_list) { @@ -3288,15 +3287,15 @@ static int __get_nat_bitmaps(struct f2fs_sb_info *sbi) nat_bits_addr = __start_cp_addr(sbi) + BLKS_PER_SEG(sbi) - nm_i->nat_bits_blocks; for (i = 0; i < nm_i->nat_bits_blocks; i++) { - struct folio *folio; + struct f2fs_cached_block *entry; - folio = f2fs_get_meta_folio(sbi, nat_bits_addr++); - if (IS_ERR(folio)) - return PTR_ERR(folio); + entry = f2fs_get_meta_cache(sbi, nat_bits_addr++); + if (IS_ERR(entry)) + return PTR_ERR(entry); memcpy(nm_i->nat_bits + F2FS_BLK_TO_BYTES(i), - folio_address(folio), F2FS_BLKSIZE); - f2fs_folio_put(folio, true); + cache_address(entry), F2FS_BLKSIZE); + f2fs_put_cache(entry, true); } cp_ver |= (cur_cp_crc(ckpt) << 32); diff --git a/fs/f2fs/node.h b/fs/f2fs/node.h index 5e114f352099..de8dcd5d4392 100644 --- a/fs/f2fs/node.h +++ b/fs/f2fs/node.h @@ -311,9 +311,9 @@ static inline void fill_node_footer_blkaddr(struct folio *folio, block_t blkaddr rn->footer.next_blkaddr = cpu_to_le32(blkaddr); } -static inline bool is_recoverable_dnode(const struct folio *folio) +static inline bool is_recoverable_dnode(struct f2fs_sb_info *sbi, const struct folio *folio) { - struct f2fs_checkpoint *ckpt = F2FS_CKPT(F2FS_F_SB(folio)); + struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); __u64 cp_ver = cur_cp_version(ckpt); /* Don't care crc part, if fsck.f2fs sets it. */ diff --git a/fs/f2fs/recovery.c b/fs/f2fs/recovery.c index aaa5227739c8..5cb56b4c1879 100644 --- a/fs/f2fs/recovery.c +++ b/fs/f2fs/recovery.c @@ -182,33 +182,33 @@ static const char *recover_printable_name(struct inode *inode, return raw->i_name; } -static int recover_dentry(struct inode *inode, struct folio *ifolio, +static int recover_dentry(struct inode *inode, struct f2fs_cached_block *entry, struct list_head *dir_list) { - struct f2fs_inode *raw_inode = F2FS_INODE(ifolio); + struct f2fs_inode *raw_inode = &CACHED_NODE(entry)->i; nid_t pino = le32_to_cpu(raw_inode->i_pino); struct f2fs_dir_entry *de; struct f2fs_filename fname; struct qstr usr_fname; struct folio *folio; struct inode *dir, *einode; - struct fsync_inode_entry *entry; + struct fsync_inode_entry *fsync_entry; int err = 0; const char *name; int name_len; - entry = get_fsync_inode(dir_list, pino); - if (!entry) { - entry = add_fsync_inode(F2FS_I_SB(inode), dir_list, + fsync_entry = get_fsync_inode(dir_list, pino); + if (!fsync_entry) { + fsync_entry = add_fsync_inode(F2FS_I_SB(inode), dir_list, pino, false); - if (IS_ERR(entry)) { - dir = ERR_CAST(entry); - err = PTR_ERR(entry); + if (IS_ERR(fsync_entry)) { + dir = ERR_CAST(fsync_entry); + err = PTR_ERR(fsync_entry); goto out; } } - dir = entry->inode; + dir = fsync_entry->inode; err = init_recovered_filename(dir, inode, raw_inode, &fname, &usr_fname); if (err) goto out; @@ -256,14 +256,14 @@ static int recover_dentry(struct inode *inode, struct folio *ifolio, out: name = recover_printable_name(inode, raw_inode, &name_len); f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, dir = %llu, err = %d", - __func__, ino_of_node(ifolio), name_len, name, + __func__, ino_of_node(cache_folio(entry)), name_len, name, IS_ERR(dir) ? 0 : dir->i_ino, err); return err; } -static int recover_quota_data(struct inode *inode, struct folio *folio) +static int recover_quota_data(struct inode *inode, struct f2fs_cached_block *entry) { - struct f2fs_inode *raw = F2FS_INODE(folio); + struct f2fs_inode *raw = &CACHED_NODE(entry)->i; struct iattr attr; uid_t i_uid = le32_to_cpu(raw->i_uid); gid_t i_gid = le32_to_cpu(raw->i_gid); @@ -300,9 +300,9 @@ static void recover_inline_flags(struct inode *inode, struct f2fs_inode *ri) clear_inode_flag(inode, FI_DATA_EXIST); } -static int recover_inode(struct inode *inode, struct folio *folio) +static int recover_inode(struct inode *inode, struct f2fs_cached_block *entry) { - struct f2fs_inode *raw = F2FS_INODE(folio); + struct f2fs_inode *raw = &CACHED_NODE(entry)->i; struct f2fs_inode_info *fi = F2FS_I(inode); const char *name; int name_len; @@ -310,7 +310,7 @@ static int recover_inode(struct inode *inode, struct folio *folio) inode->i_mode = le16_to_cpu(raw->i_mode); - err = recover_quota_data(inode, folio); + err = recover_quota_data(inode, entry); if (err) return err; @@ -357,7 +357,7 @@ static int recover_inode(struct inode *inode, struct folio *folio) name = recover_printable_name(inode, raw, &name_len); f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, inline = %x", - __func__, ino_of_node(folio), name_len, name, + __func__, ino_of_node(cache_folio(entry)), name_len, name, raw->i_inline); return 0; } @@ -386,30 +386,30 @@ static int sanity_check_node_chain(struct f2fs_sb_info *sbi, block_t blkaddr, return 0; for (i = 0; i < 2; i++) { - struct folio *folio; + struct f2fs_cached_block *entry; if (!f2fs_is_valid_blkaddr(sbi, *blkaddr_fast, META_POR)) { *is_detecting = false; return 0; } - folio = f2fs_get_tmp_folio(sbi, *blkaddr_fast); - if (IS_ERR(folio)) - return PTR_ERR(folio); + entry = f2fs_get_tmp_cache(sbi, *blkaddr_fast); + if (IS_ERR(entry)) + return PTR_ERR(entry); - if (!is_recoverable_dnode(folio)) { - f2fs_folio_put(folio, true); + if (!is_recoverable_dnode(sbi, cache_folio(entry))) { + f2fs_put_cache(entry, true); *is_detecting = false; return 0; } ra_blocks = adjust_por_ra_blocks(sbi, ra_blocks, *blkaddr_fast, - next_blkaddr_of_node(folio)); + next_blkaddr_of_node(cache_folio(entry))); - *blkaddr_fast = next_blkaddr_of_node(folio); - f2fs_folio_put(folio, true); + *blkaddr_fast = next_blkaddr_of_node(cache_folio(entry)); + f2fs_put_cache(entry, true); - f2fs_ra_meta_pages_cond(sbi, *blkaddr_fast, ra_blocks); + f2fs_ra_meta_caches_cond(sbi, *blkaddr_fast, ra_blocks); } if (*blkaddr_fast == blkaddr) { @@ -434,45 +434,47 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head, blkaddr_fast = blkaddr; while (1) { - struct fsync_inode_entry *entry; - struct folio *folio; + struct fsync_inode_entry *fsync_entry; + struct f2fs_cached_block *entry; + struct f2fs_node *rn; if (!f2fs_is_valid_blkaddr(sbi, blkaddr, META_POR)) return 0; - folio = f2fs_get_tmp_folio(sbi, blkaddr); - if (IS_ERR(folio)) { - err = PTR_ERR(folio); + entry = f2fs_get_tmp_cache(sbi, blkaddr); + if (IS_ERR(entry)) { + err = PTR_ERR(entry); break; } + rn = CACHED_NODE(entry); - if (!is_recoverable_dnode(folio)) { - f2fs_folio_put(folio, true); + if (!is_recoverable_dnode(sbi, cache_folio(entry))) { + f2fs_put_cache(entry, true); break; } - if (!is_fsync_dnode(folio)) + if (!is_fsync_dnode(cache_folio(entry))) goto next; - entry = get_fsync_inode(head, ino_of_node(folio)); - if (!entry) { + fsync_entry = get_fsync_inode(head, ino_of_node(cache_folio(entry))); + if (!fsync_entry) { bool quota_inode = false; if (!check_only && - IS_INODE(folio) && - is_dent_dnode(folio)) { - err = f2fs_recover_inode_page(sbi, folio); + IS_INODE(cache_folio(entry)) && + is_dent_dnode(cache_folio(entry))) { + err = f2fs_recover_inode_page(sbi, entry); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); break; } quota_inode = true; } - entry = add_fsync_inode(sbi, head, ino_of_node(folio), + fsync_entry = add_fsync_inode(sbi, head, ino_of_node(cache_folio(entry)), quota_inode); - if (IS_ERR(entry)) { - err = PTR_ERR(entry); + if (IS_ERR(fsync_entry)) { + err = PTR_ERR(fsync_entry); /* * CP | dnode(F) | inode(DF) * For this case, we should not give up now. @@ -482,18 +484,18 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head, *new_inode = true; goto next; } - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); break; } } - entry->blkaddr = blkaddr; + fsync_entry->blkaddr = blkaddr; - if (IS_INODE(folio) && is_dent_dnode(folio)) - entry->last_dentry = blkaddr; + if (IS_INODE(cache_folio(entry)) && is_dent_dnode(cache_folio(entry))) + fsync_entry->last_dentry = blkaddr; next: /* check next segment */ - blkaddr = next_blkaddr_of_node(folio); - f2fs_folio_put(folio, true); + blkaddr = next_blkaddr_of_node(cache_folio(entry)); + f2fs_put_cache(entry, true); err = sanity_check_node_chain(sbi, blkaddr, &blkaddr_fast, &is_detecting); @@ -519,7 +521,8 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, unsigned short blkoff = GET_BLKOFF_FROM_SEG0(sbi, blkaddr); struct f2fs_summary_block *sum_node; struct f2fs_summary sum; - struct folio *sum_folio, *node_folio; + struct f2fs_cached_block *entry = NULL; + struct folio *node_folio; struct dnode_of_data tdn = *dn; nid_t ino, nid; struct inode *inode; @@ -541,12 +544,12 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, } } - sum_folio = f2fs_get_sum_folio(sbi, segno); - if (IS_ERR(sum_folio)) - return PTR_ERR(sum_folio); - sum_node = SUM_BLK_PAGE_ADDR(sbi, sum_folio, segno); + entry = f2fs_get_sum_cache(sbi, segno); + if (IS_ERR(entry)) + return PTR_ERR(entry); + sum_node = SUM_BLK_ENTRY_ADDR(sbi, entry, segno); sum = sum_entries(sum_node)[blkoff]; - f2fs_folio_put(sum_folio, true); + f2fs_put_cache(entry, true); got_it: /* Use the locked dnode page and inode */ nid = le32_to_cpu(sum.nid); @@ -645,7 +648,7 @@ static int f2fs_reserve_new_block_retry(struct dnode_of_data *dn) } static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, - struct folio *folio) + struct f2fs_cached_block *entry) { struct dnode_of_data dn; struct node_info ni; @@ -653,19 +656,19 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, int err = 0, recovered = 0; /* step 1: recover xattr */ - if (IS_INODE(folio)) { - err = f2fs_recover_inline_xattr(inode, folio); + if (IS_INODE(cache_folio(entry))) { + err = f2fs_recover_inline_xattr(inode, entry); if (err) goto out; - } else if (f2fs_has_xattr_block(ofs_of_node(folio))) { - err = f2fs_recover_xattr_data(inode, folio); + } else if (f2fs_has_xattr_block(ofs_of_node(cache_folio(entry)))) { + err = f2fs_recover_xattr_data(inode, entry); if (!err) recovered++; goto out; } /* step 2: recover inline data */ - err = f2fs_recover_inline_data(inode, folio); + err = f2fs_recover_inline_data(inode, entry); if (err) { if (err == 1) err = 0; @@ -673,8 +676,8 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, } /* step 3: recover data indices */ - start = f2fs_start_bidx_of_node(ofs_of_node(folio), inode); - end = start + ADDRS_PER_PAGE(folio, inode); + start = f2fs_start_bidx_of_node(ofs_of_node(cache_folio(entry)), inode); + end = start + addrs_per_page(inode, IS_INODE(cache_folio(entry))); set_new_dnode(&dn, inode, NULL, NULL, 0); retry_dn: @@ -693,12 +696,12 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, if (err) goto err; - f2fs_bug_on(sbi, ni.ino != ino_of_node(folio)); + f2fs_bug_on(sbi, ni.ino != ino_of_node(cache_folio(entry))); - if (ofs_of_node(dn.node_folio) != ofs_of_node(folio)) { + if (ofs_of_node(dn.node_folio) != ofs_of_node(cache_folio(entry))) { f2fs_warn(sbi, "Inconsistent ofs_of_node, ino:%llu, ofs:%u, %u", inode->i_ino, ofs_of_node(dn.node_folio), - ofs_of_node(folio)); + ofs_of_node(cache_folio(entry))); err = -EFSCORRUPTED; f2fs_handle_error(sbi, ERROR_INCONSISTENT_FOOTER); fserror_report_file_metadata(dn.inode, err, GFP_NOFS); @@ -709,7 +712,7 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, block_t src, dest; src = f2fs_data_blkaddr(&dn); - dest = data_blkaddr(dn.inode, folio, dn.ofs_in_node); + dest = data_blkaddr(dn.inode, cache_folio(entry), dn.ofs_in_node); if (__is_valid_data_blkaddr(src) && !f2fs_is_valid_blkaddr(sbi, src, META_POR)) { @@ -784,16 +787,16 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, } } - copy_node_footer(dn.node_folio, folio); + copy_node_footer(dn.node_folio, cache_folio(entry)); fill_node_footer(dn.node_folio, dn.nid, ni.ino, - ofs_of_node(folio), false); + ofs_of_node(cache_folio(entry)), false); folio_mark_dirty(dn.node_folio); err: f2fs_put_dnode(&dn); out: f2fs_notice(sbi, "recover_data: ino = %llx, nid = %x (i_size: %s), " "range (%u, %u), recovered = %d, err = %d", - inode->i_ino, nid_of_node(folio), + inode->i_ino, nid_of_node(cache_folio(entry)), file_keep_isize(inode) ? "keep" : "recover", start, end, recovered, err); return err; @@ -820,26 +823,26 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list, blkaddr = NEXT_FREE_BLKADDR(sbi, curseg); while (1) { - struct fsync_inode_entry *entry; - struct folio *folio; + struct fsync_inode_entry *fsync_entry; + struct f2fs_cached_block *entry; if (!f2fs_is_valid_blkaddr(sbi, blkaddr, META_POR)) break; - folio = f2fs_get_tmp_folio(sbi, blkaddr); - if (IS_ERR(folio)) { - err = PTR_ERR(folio); + entry = f2fs_get_tmp_cache(sbi, blkaddr); + if (IS_ERR(entry)) { + err = PTR_ERR(entry); break; } - if (!is_recoverable_dnode(folio)) { - f2fs_folio_put(folio, true); + if (!is_recoverable_dnode(sbi, cache_folio(entry))) { + f2fs_put_cache(entry, true); break; } recoverable_dnode++; - entry = get_fsync_inode(inode_list, ino_of_node(folio)); - if (!entry) + fsync_entry = get_fsync_inode(inode_list, ino_of_node(cache_folio(entry))); + if (!fsync_entry) goto next; fsynced_dnode++; /* @@ -847,40 +850,40 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list, * In this case, we can lose the latest inode(x). * So, call recover_inode for the inode update. */ - if (IS_INODE(folio)) { - err = recover_inode(entry->inode, folio); + if (IS_INODE(cache_folio(entry))) { + err = recover_inode(fsync_entry->inode, entry); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); break; } recovered_inode++; } - if (entry->last_dentry == blkaddr) { - err = recover_dentry(entry->inode, folio, dir_list); + if (fsync_entry->last_dentry == blkaddr) { + err = recover_dentry(fsync_entry->inode, entry, dir_list); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); break; } recovered_dentry++; } - err = do_recover_data(sbi, entry->inode, folio); + err = do_recover_data(sbi, fsync_entry->inode, entry); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); break; } recovered_dnode++; - if (entry->blkaddr == blkaddr) - list_move_tail(&entry->list, tmp_inode_list); + if (fsync_entry->blkaddr == blkaddr) + list_move_tail(&fsync_entry->list, tmp_inode_list); next: ra_blocks = adjust_por_ra_blocks(sbi, ra_blocks, blkaddr, - next_blkaddr_of_node(folio)); + next_blkaddr_of_node(cache_folio(entry))); /* check next segment */ - blkaddr = next_blkaddr_of_node(folio); - f2fs_folio_put(folio, true); + blkaddr = next_blkaddr_of_node(cache_folio(entry)); + f2fs_put_cache(entry, true); - f2fs_ra_meta_pages_cond(sbi, blkaddr, ra_blocks); + f2fs_ra_meta_caches_cond(sbi, blkaddr, ra_blocks); total_dnode++; } if (!err) @@ -937,12 +940,11 @@ int f2fs_recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only) destroy_fsync_dnodes(&tmp_inode_list, err); /* truncate meta pages to be used by the recovery */ - truncate_inode_pages_range(META_MAPPING(sbi), - (loff_t)MAIN_BLKADDR(sbi) << PAGE_SHIFT, -1); - + f2fs_truncate_meta_caches(sbi, MAIN_BLKADDR(sbi), + MAX_BLKADDR(sbi) - MAIN_BLKADDR(sbi)); if (err) { truncate_inode_pages_final(NODE_MAPPING(sbi)); - truncate_inode_pages_final(META_MAPPING(sbi)); + f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); } /* diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c index 56decf9c691c..cbb3a8c9f4ab 100644 --- a/fs/f2fs/segment.c +++ b/fs/f2fs/segment.c @@ -2774,63 +2774,62 @@ int f2fs_npages_for_summary_flush(struct f2fs_sb_info *sbi, bool for_ra) return 3; } -/* - * Caller should put this summary folio - */ -struct folio *f2fs_get_sum_folio(struct f2fs_sb_info *sbi, unsigned int segno) + +struct f2fs_cached_block *f2fs_get_sum_cache(struct f2fs_sb_info *sbi, + unsigned int segno) { if (unlikely(f2fs_cp_error(sbi))) return ERR_PTR(-EIO); - return f2fs_get_meta_folio_retry(sbi, GET_SUM_BLOCK(sbi, segno)); + return f2fs_get_meta_cache_retry(sbi, GET_SUM_BLOCK(sbi, segno)); } void f2fs_update_meta_page(struct f2fs_sb_info *sbi, void *src, block_t blk_addr) { - struct folio *folio; + struct f2fs_cached_block *entry; if (!f2fs_sb_has_packed_ssa(sbi)) - folio = f2fs_grab_meta_folio(sbi, blk_addr); + entry = f2fs_grab_meta_cache(sbi, blk_addr); else - folio = f2fs_get_meta_folio_retry(sbi, blk_addr); + entry = f2fs_get_meta_cache_retry(sbi, blk_addr); - if (IS_ERR(folio)) + if (IS_ERR(entry)) return; - memcpy(folio_address(folio), src, PAGE_SIZE); - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); + memcpy(cache_address(entry), src, sbi->blocksize); + f2fs_mark_cache_dirty(entry); + f2fs_put_cache(entry, true); } static void write_sum_page(struct f2fs_sb_info *sbi, struct f2fs_summary_block *sum_blk, unsigned int segno) { - struct folio *folio; + struct f2fs_cached_block *entry; if (!f2fs_sb_has_packed_ssa(sbi)) return f2fs_update_meta_page(sbi, (void *)sum_blk, GET_SUM_BLOCK(sbi, segno)); - folio = f2fs_get_sum_folio(sbi, segno); - if (IS_ERR(folio)) + entry = f2fs_get_sum_cache(sbi, GET_SUM_BLOCK(sbi, segno)); + if (IS_ERR(entry)) return; - memcpy(SUM_BLK_PAGE_ADDR(sbi, folio, segno), sum_blk, + memcpy(SUM_BLK_ENTRY_ADDR(sbi, entry, segno), sum_blk, sbi->sum_blocksize); - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); + f2fs_mark_cache_dirty(entry); + f2fs_put_cache(entry, true); } static void write_current_sum_page(struct f2fs_sb_info *sbi, int type, block_t blk_addr) { struct curseg_info *curseg = CURSEG_I(sbi, type); - struct folio *folio = f2fs_grab_meta_folio(sbi, blk_addr); + struct f2fs_cached_block *entry = f2fs_grab_meta_cache(sbi, blk_addr); struct f2fs_summary_block *src = curseg->sum_blk; struct f2fs_summary_block *dst; - dst = folio_address(folio); - memset(dst, 0, PAGE_SIZE); + dst = cache_address(entry); + memset(dst, 0, sbi->blocksize); mutex_lock(&curseg->curseg_mutex); @@ -2843,8 +2842,8 @@ static void write_current_sum_page(struct f2fs_sb_info *sbi, mutex_unlock(&curseg->curseg_mutex); - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); + f2fs_mark_cache_dirty(entry); + f2fs_put_cache(entry, true); } static int is_next_segment_free(struct f2fs_sb_info *sbi, @@ -3160,7 +3159,7 @@ static int change_curseg(struct f2fs_sb_info *sbi, int type) struct curseg_info *curseg = CURSEG_I(sbi, type); unsigned int new_segno = curseg->next_segno; struct f2fs_summary_block *sum_node; - struct folio *sum_folio; + struct f2fs_cached_block *entry = NULL; if (curseg->inited) write_sum_page(sbi, curseg->sum_blk, curseg->segno); @@ -3176,15 +3175,15 @@ static int change_curseg(struct f2fs_sb_info *sbi, int type) curseg->alloc_type = SSR; curseg->next_blkoff = __next_free_blkoff(sbi, curseg->segno, 0); - sum_folio = f2fs_get_sum_folio(sbi, new_segno); - if (IS_ERR(sum_folio)) { + entry = f2fs_get_sum_cache(sbi, new_segno); + if (IS_ERR(entry)) { /* GC won't be able to use stale summary pages by cp_error */ memset(curseg->sum_blk, 0, sbi->sum_entry_size); - return PTR_ERR(sum_folio); + return PTR_ERR(entry); } - sum_node = SUM_BLK_PAGE_ADDR(sbi, sum_folio, new_segno); + sum_node = SUM_BLK_ENTRY_ADDR(sbi, entry, new_segno); memcpy(curseg->sum_blk, sum_node, sbi->sum_entry_size); - f2fs_folio_put(sum_folio, true); + f2fs_put_cache(entry, true); return 0; } @@ -4127,8 +4126,9 @@ static void do_write_page(struct f2fs_summary *sum, struct f2fs_io_info *fio) f2fs_up_read(&fio->sbi->io_order_lock); } -void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct folio *folio, - enum iostat_type io_type) +void f2fs_do_write_meta_cache(struct f2fs_sb_info *sbi, + struct f2fs_cached_block *entry, + enum iostat_type io_type) { struct f2fs_io_info fio = { .sbi = sbi, @@ -4136,20 +4136,21 @@ void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct folio *folio, .temp = HOT, .op = REQ_OP_WRITE, .op_flags = REQ_SYNC | REQ_META | REQ_PRIO, - .old_blkaddr = folio->index, - .new_blkaddr = folio->index, - .folio = folio, + .old_blkaddr = entry->index, + .new_blkaddr = entry->index, .encrypted_page = NULL, .in_list = 0, + .cache_entry = entry, + .is_cache = 1, }; - if (unlikely(folio->index >= MAIN_BLKADDR(sbi))) + if (unlikely(entry->index >= MAIN_BLKADDR(sbi))) fio.op_flags &= ~REQ_META; - folio_start_writeback(folio); - f2fs_submit_page_write(&fio); + f2fs_start_cache_writeback(entry); + f2fs_submit_cache_write(&fio); - stat_inc_meta_count(sbi, folio->index); + stat_inc_meta_count(sbi, entry->index); f2fs_update_iostat(sbi, NULL, io_type, F2FS_BLKSIZE); } @@ -4206,7 +4207,7 @@ int f2fs_inplace_write_data(struct f2fs_io_info *fio) } if (fio->meta_gc) - f2fs_truncate_meta_inode_pages(sbi, fio->new_blkaddr, 1); + f2fs_truncate_meta_caches(sbi, fio->new_blkaddr, 1); stat_inc_inplace_blocks(fio->sbi); @@ -4372,7 +4373,7 @@ void f2fs_folio_wait_writeback(struct folio *folio, enum page_type type, void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - struct folio *cfolio; + struct f2fs_cached_block *entry; if (!f2fs_meta_inode_gc_required(inode)) return; @@ -4380,11 +4381,12 @@ void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr) if (!__is_valid_data_blkaddr(blkaddr)) return; - cfolio = filemap_lock_folio(META_MAPPING(sbi), blkaddr); - if (!IS_ERR(cfolio)) { - f2fs_folio_wait_writeback(cfolio, DATA, true, true); - f2fs_folio_put(cfolio, true); - } + entry = f2fs_find_cache(META_CACHE(sbi), blkaddr); + if (IS_ERR(entry)) + return; + f2fs_lock_cache(entry); + f2fs_cache_wait_writeback_cond(entry, DATA); + f2fs_put_cache(entry, true); } void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr, @@ -4399,7 +4401,7 @@ void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr, for (i = 0; i < len; i++) f2fs_wait_on_block_writeback(inode, blkaddr + i); - f2fs_truncate_meta_inode_pages(sbi, blkaddr, len); + f2fs_truncate_meta_caches(sbi, blkaddr, len); } static int read_compacted_summaries(struct f2fs_sb_info *sbi) @@ -4407,16 +4409,16 @@ static int read_compacted_summaries(struct f2fs_sb_info *sbi) struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); struct curseg_info *seg_i; unsigned char *kaddr; - struct folio *folio; + struct f2fs_cached_block *entry; block_t start; int i, j, offset; start = start_sum_block(sbi); - folio = f2fs_get_meta_folio(sbi, start++); - if (IS_ERR(folio)) - return PTR_ERR(folio); - kaddr = folio_address(folio); + entry = f2fs_get_meta_cache(sbi, start++); + if (IS_ERR(entry)) + return PTR_ERR(entry); + kaddr = cache_address(entry); /* Step 1: restore nat cache */ seg_i = CURSEG_I(sbi, CURSEG_HOT_DATA); @@ -4453,16 +4455,16 @@ static int read_compacted_summaries(struct f2fs_sb_info *sbi) SUM_FOOTER_SIZE) continue; - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); - folio = f2fs_get_meta_folio(sbi, start++); - if (IS_ERR(folio)) - return PTR_ERR(folio); - kaddr = folio_address(folio); + entry = f2fs_get_meta_cache(sbi, start++); + if (IS_ERR(entry)) + return PTR_ERR(entry); + kaddr = cache_address(entry); offset = 0; } } - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); return 0; } @@ -4471,7 +4473,7 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type) struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); struct f2fs_summary_block *sum; struct curseg_info *curseg; - struct folio *new; + struct f2fs_cached_block *entry; unsigned short blk_off; unsigned int segno = 0; block_t blk_addr = 0; @@ -4498,10 +4500,10 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type) blk_addr = GET_SUM_BLOCK(sbi, segno); } - new = f2fs_get_meta_folio(sbi, blk_addr); - if (IS_ERR(new)) - return PTR_ERR(new); - sum = folio_address(new); + entry = f2fs_get_meta_cache(sbi, blk_addr); + if (IS_ERR(entry)) + return PTR_ERR(entry); + sum = cache_address(entry); if (IS_NODESEG(type)) { if (__exist_node_summaries(sbi)) { @@ -4538,7 +4540,7 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type) curseg->next_blkoff = blk_off; mutex_unlock(&curseg->curseg_mutex); out: - f2fs_folio_put(new, true); + f2fs_put_cache(entry, true); return err; } @@ -4553,8 +4555,8 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi) int npages = f2fs_npages_for_summary_flush(sbi, true); if (npages >= 2) - f2fs_ra_meta_pages(sbi, start_sum_block(sbi), npages, - META_CP, true); + f2fs_ra_meta_caches(sbi, start_sum_block(sbi), + npages, META_CP, true); /* restore for compacted data summary */ err = read_compacted_summaries(sbi); @@ -4564,7 +4566,7 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi) } if (__exist_node_summaries(sbi)) - f2fs_ra_meta_pages(sbi, + f2fs_ra_meta_caches(sbi, sum_blk_addr(sbi, NR_CURSEG_PERSIST_TYPE, type), NR_CURSEG_PERSIST_TYPE - type, META_CP, true); @@ -4587,16 +4589,16 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi) static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr) { - struct folio *folio; + struct f2fs_cached_block *entry = NULL; unsigned char *kaddr; struct f2fs_summary *summary; struct curseg_info *seg_i; int written_size = 0; int i, j; - folio = f2fs_grab_meta_folio(sbi, blkaddr++); - kaddr = folio_address(folio); - memset(kaddr, 0, PAGE_SIZE); + entry = f2fs_grab_meta_cache(sbi, blkaddr++); + kaddr = cache_address(entry); + memset(kaddr, 0, sbi->blocksize); /* Step 1: write nat cache */ seg_i = CURSEG_I(sbi, CURSEG_HOT_DATA); @@ -4612,10 +4614,10 @@ static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr) for (i = CURSEG_HOT_DATA; i <= CURSEG_COLD_DATA; i++) { seg_i = CURSEG_I(sbi, i); for (j = 0; j < f2fs_curseg_valid_blocks(sbi, i); j++) { - if (!folio) { - folio = f2fs_grab_meta_folio(sbi, blkaddr++); - kaddr = folio_address(folio); - memset(kaddr, 0, PAGE_SIZE); + if (!entry) { + entry = f2fs_grab_meta_cache(sbi, blkaddr++); + kaddr = cache_address(entry); + memset(kaddr, 0, sbi->blocksize); written_size = 0; } summary = (struct f2fs_summary *)(kaddr + written_size); @@ -4626,14 +4628,14 @@ static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr) SUM_FOOTER_SIZE) continue; - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); - folio = NULL; + f2fs_mark_cache_dirty(entry); + f2fs_put_cache(entry, true); + entry = NULL; } } - if (folio) { - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); + if (entry) { + f2fs_mark_cache_dirty(entry); + f2fs_put_cache(entry, true); } } @@ -4687,29 +4689,29 @@ int f2fs_lookup_journal_in_cursum(struct f2fs_sb_info *sbi, return -1; } -static struct folio *get_current_sit_folio(struct f2fs_sb_info *sbi, +static struct f2fs_cached_block *get_current_sit_cache(struct f2fs_sb_info *sbi, unsigned int segno) { - return f2fs_get_meta_folio(sbi, current_sit_addr(sbi, segno)); + return f2fs_get_meta_cache(sbi, current_sit_addr(sbi, segno)); } -static struct folio *get_next_sit_folio(struct f2fs_sb_info *sbi, +static struct f2fs_cached_block *get_next_sit_cache(struct f2fs_sb_info *sbi, unsigned int start) { struct sit_info *sit_i = SIT_I(sbi); - struct folio *folio; + struct f2fs_cached_block *entry; pgoff_t src_off, dst_off; src_off = current_sit_addr(sbi, start); dst_off = next_sit_addr(sbi, src_off); - folio = f2fs_grab_meta_folio(sbi, dst_off); - seg_info_to_sit_folio(sbi, folio, start); + entry = f2fs_grab_meta_cache(sbi, dst_off); + seg_info_to_sit_block(sbi, entry, start); - folio_mark_dirty(folio); + f2fs_mark_cache_dirty(entry); set_to_next_sit(sit_i, start); - return folio; + return entry; } static struct sit_entry_set *grab_sit_entry_set(void) @@ -4839,8 +4841,9 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) * #2, flush sit entries to sit page. */ list_for_each_entry_safe(ses, tmp, head, set_list) { - struct folio *folio = NULL; + struct f2fs_cached_block *entry = NULL; struct f2fs_sit_block *raw_sit = NULL; + unsigned int start_segno = ses->start_segno; unsigned int end = min(start_segno + SIT_ENTRY_PER_BLOCK, (unsigned long)MAIN_SEGS(sbi)); @@ -4854,8 +4857,8 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) if (to_journal) { down_write(&curseg->journal_rwsem); } else { - folio = get_next_sit_folio(sbi, start_segno); - raw_sit = folio_address(folio); + entry = get_next_sit_cache(sbi, start_segno); + raw_sit = cache_address(entry); } /* flush dirty sit entries in region of current sit set */ @@ -4900,7 +4903,7 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) if (to_journal) up_write(&curseg->journal_rwsem); else - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); f2fs_bug_on(sbi, ses->entry_cnt); release_sit_entry_set(ses); @@ -5091,7 +5094,7 @@ static int build_sit_entries(struct f2fs_sb_info *sbi) block_t sit_valid_blocks[2] = {0, 0}; do { - readed = f2fs_ra_meta_pages(sbi, start_blk, BIO_MAX_VECS, + readed = f2fs_ra_meta_caches(sbi, start_blk, BIO_MAX_VECS, META_SIT, true); start = start_blk * sit_i->sents_per_block; @@ -5099,15 +5102,15 @@ static int build_sit_entries(struct f2fs_sb_info *sbi) for (; start < end && start < MAIN_SEGS(sbi); start++) { struct f2fs_sit_block *sit_blk; - struct folio *folio; + struct f2fs_cached_block *entry; se = &sit_i->sentries[start]; - folio = get_current_sit_folio(sbi, start); - if (IS_ERR(folio)) - return PTR_ERR(folio); - sit_blk = folio_address(folio); + entry = get_current_sit_cache(sbi, start); + if (IS_ERR(entry)) + return PTR_ERR(entry); + sit_blk = cache_address(entry); sit = sit_blk->entries[SIT_ENTRY_OFFSET(sit_i, start)]; - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); err = check_block_count(sbi, start, &sit); if (err) diff --git a/fs/f2fs/segment.h b/fs/f2fs/segment.h index db1079169a23..fff35c63a00f 100644 --- a/fs/f2fs/segment.h +++ b/fs/f2fs/segment.h @@ -95,6 +95,8 @@ static inline void sanity_check_seg_type(struct f2fs_sb_info *sbi, #define GET_SUM_BLKOFF(sbi, segno) (segno % (sbi)->sums_per_block) #define SUM_BLK_PAGE_ADDR(sbi, folio, segno) \ (folio_address(folio) + GET_SUM_BLKOFF(sbi, segno) * (sbi)->sum_blocksize) +#define SUM_BLK_ENTRY_ADDR(sbi, entry, segno) \ + (cache_address(entry) + GET_SUM_BLKOFF(sbi, segno) * (sbi)->sum_blocksize) #define GET_SUM_TYPE(footer) ((footer)->entry_type) #define SET_SUM_TYPE(footer, type) ((footer)->entry_type = (type)) @@ -417,8 +419,8 @@ static inline void __seg_info_to_raw_sit(struct seg_entry *se, rs->mtime = cpu_to_le64(se->mtime); } -static inline void seg_info_to_sit_folio(struct f2fs_sb_info *sbi, - struct folio *folio, unsigned int start) +static inline void seg_info_to_sit_block(struct f2fs_sb_info *sbi, + struct f2fs_cached_block *entry, unsigned int start) { struct f2fs_sit_block *raw_sit; struct seg_entry *se; @@ -427,8 +429,8 @@ static inline void seg_info_to_sit_folio(struct f2fs_sb_info *sbi, (unsigned long)MAIN_SEGS(sbi)); int i; - raw_sit = folio_address(folio); - memset(raw_sit, 0, PAGE_SIZE); + raw_sit = cache_address(entry); + memset(raw_sit, 0, sbi->blocksize); for (i = 0; i < end - start; i++) { rs = &raw_sit->entries[i]; se = get_seg_entry(sbi, start + i); @@ -991,6 +993,27 @@ static inline int nr_pages_to_skip(struct f2fs_sb_info *sbi, int type) return 0; } +/* + * When writing cache asynchronously, align nr_to_write to BIO_MAX_VECS. + */ +static inline long adjust_flush_cache_number(struct f2fs_sb_info *sbi, int type) +{ + long nr_to_write; + + switch (type) { + case META: + nr_to_write = BIO_MAX_VECS; + break; + case NODE: + nr_to_write = BIO_MAX_VECS << 1; + break; + default: + f2fs_bug_on(sbi, 1); + return 0; + } + return nr_to_write; +} + /* * When writing pages, it'd better align nr_to_write for segment size. */ diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index 89affe72f4fc..d3dee9b7d999 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -1839,8 +1839,7 @@ static int f2fs_drop_inode(struct inode *inode) * drop useless meta/node dirty pages. */ if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) { - if (inode->i_ino == F2FS_NODE_INO(sbi) || - inode->i_ino == F2FS_META_INO(sbi)) { + if (inode->i_ino == F2FS_NODE_INO(sbi)) { trace_f2fs_drop_inode(inode, 1); return 1; } @@ -1942,8 +1941,7 @@ static void f2fs_dirty_inode(struct inode *inode, int flags) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - if (inode->i_ino == F2FS_NODE_INO(sbi) || - inode->i_ino == F2FS_META_INO(sbi)) + if (inode->i_ino == F2FS_NODE_INO(sbi)) return; if (is_inode_flag_set(inode, FI_AUTO_RECOVER)) @@ -2039,9 +2037,10 @@ static void f2fs_put_super(struct super_block *sb) f2fs_wait_on_all_pages(sbi, F2FS_WB_CP_DATA); - if (err || f2fs_cp_error(sbi)) { + if (err || f2fs_cp_error(sbi) || + unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) { truncate_inode_pages_final(NODE_MAPPING(sbi)); - truncate_inode_pages_final(META_MAPPING(sbi)); + f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); } f2fs_bug_on(sbi, sbi->fsync_node_num); @@ -2051,9 +2050,6 @@ static void f2fs_put_super(struct super_block *sb) iput(sbi->node_inode); sbi->node_inode = NULL; - iput(sbi->meta_inode); - sbi->meta_inode = NULL; - f2fs_destroy_cache(META_CACHE(sbi)); /* Should check the page counts after dropping all node/meta pages */ @@ -4383,7 +4379,6 @@ static void init_sb_info(struct f2fs_sb_info *sbi) sbi->allocate_section_policy = ALLOCATE_FORWARD_NOHINT; F2FS_ROOT_INO(sbi) = le32_to_cpu(raw_super->root_ino); F2FS_NODE_INO(sbi) = le32_to_cpu(raw_super->node_ino); - F2FS_META_INO(sbi) = le32_to_cpu(raw_super->meta_ino); sbi->cur_victim_sec = NULL_SECNO; sbi->gc_mode = GC_NORMAL; sbi->next_victim_seg[BG_GC] = NULL_SEGNO; @@ -5224,18 +5219,10 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) if (err) goto free_page_array_cache; - /* get an inode for meta space */ - sbi->meta_inode = f2fs_iget(sb, F2FS_META_INO(sbi)); - if (IS_ERR(sbi->meta_inode)) { - f2fs_err(sbi, "Failed to read F2FS meta data inode"); - err = PTR_ERR(sbi->meta_inode); - goto free_meta_cache; - } - err = f2fs_get_valid_checkpoint(sbi); if (err) { f2fs_err(sbi, "Failed to get valid F2FS checkpoint"); - goto free_meta_inode; + goto free_meta_cache; } if (__is_set_ckpt_flags(F2FS_CKPT(sbi), CP_QUOTA_NEED_FSCK_FLAG)) @@ -5529,7 +5516,7 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) * followed by f2fs_write_checkpoint() through f2fs_write_node_pages(), which * falls into an infinite loop in f2fs_sync_meta_pages(). */ - truncate_inode_pages_final(META_MAPPING(sbi)); + f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); /* evict some inodes being cached by GC */ evict_inodes(sb); f2fs_unregister_sysfs(sbi); @@ -5559,10 +5546,6 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) free_devices: destroy_device_list(sbi); kvfree(sbi->ckpt); -free_meta_inode: - make_bad_inode(sbi->meta_inode); - iput(sbi->meta_inode); - sbi->meta_inode = NULL; free_meta_cache: f2fs_destroy_cache(META_CACHE(sbi)); free_page_array_cache: diff --git a/include/linux/f2fs_fs.h b/include/linux/f2fs_fs.h index bb2b6cd5d507..0c027d00a1ea 100644 --- a/include/linux/f2fs_fs.h +++ b/include/linux/f2fs_fs.h @@ -35,7 +35,6 @@ #define F2FS_ROOT_INO(sbi) ((sbi)->root_ino_num) #define F2FS_NODE_INO(sbi) ((sbi)->node_ino_num) -#define F2FS_META_INO(sbi) ((sbi)->meta_ino_num) #define F2FS_COMPRESS_INO(sbi) (NM_I(sbi)->max_nid) #define F2FS_MAX_QUOTAS 3 -- 2.49.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [PATCH v1 05/12] f2fs: cache: use meta cache @ 2026-08-20 3:17 ` Chao Yu 0 siblings, 0 replies; 34+ messages in thread From: Chao Yu @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-f2fs-devel, linux-kernel, Chao Yu This patch migrates F2FS meta block caching from the fake VFS inode page cache (sbi->meta_inode) to meta cache (sbi->meta_blocks). It converts CP, SIT, NAT, SSA, recovery, and GC metadata I/O paths to operate on struct f2fs_cached_block instead of folio, and removes sbi->meta_inode. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 2 + fs/f2fs/checkpoint.c | 394 ++++++++++++++++++---------------------- fs/f2fs/compress.c | 4 +- fs/f2fs/data.c | 29 ++- fs/f2fs/debug.c | 12 +- fs/f2fs/f2fs.h | 81 +++------ fs/f2fs/file.c | 4 +- fs/f2fs/gc.c | 134 +++++++------- fs/f2fs/inline.c | 9 +- fs/f2fs/inode.c | 9 +- fs/f2fs/node.c | 125 +++++++------ fs/f2fs/node.h | 4 +- fs/f2fs/recovery.c | 198 ++++++++++---------- fs/f2fs/segment.c | 201 ++++++++++---------- fs/f2fs/segment.h | 31 +++- fs/f2fs/super.c | 31 +--- include/linux/f2fs_fs.h | 1 - 17 files changed, 606 insertions(+), 663 deletions(-) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index cb5b26046162..afef808e485a 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -631,6 +631,8 @@ static int f2fs_cache_writeback_kthread(void *data) break; if (f2fs_cp_error(sbi)) continue; + + f2fs_write_meta_caches(sbi); } return 0; } diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c index 729d19680caf..1a7083540b82 100644 --- a/fs/f2fs/checkpoint.c +++ b/fs/f2fs/checkpoint.c @@ -17,6 +17,7 @@ #include <linux/delayacct.h> #include <linux/ioprio.h> #include <linux/math64.h> +#include <linux/freezer.h> #include "f2fs.h" #include "node.h" @@ -235,27 +236,27 @@ struct kmem_cache *f2fs_inode_entry_slab; /* * We guarantee no failure on the returned page. */ -struct folio *f2fs_grab_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index) +struct f2fs_cached_block *f2fs_grab_meta_cache(struct f2fs_sb_info *sbi, + pgoff_t index) { - struct address_space *mapping = META_MAPPING(sbi); - struct folio *folio; + struct f2fs_cached_block *entry; repeat: - folio = f2fs_grab_cache_folio(mapping, index, false); - if (IS_ERR(folio)) { + entry = f2fs_grab_cache(META_CACHE(sbi), index, + F2FS_CACHE_LOCK_CREATE); + if (IS_ERR(entry)) { cond_resched(); goto repeat; } - f2fs_folio_wait_writeback(folio, META, true, true); - if (!folio_test_uptodate(folio)) - folio_mark_uptodate(folio); - return folio; + f2fs_cache_wait_writeback(entry); + if (!f2fs_cache_test_uptodate(entry)) + f2fs_cache_set_uptodate(entry); + return entry; } -static struct folio *__get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index, - bool is_meta) +static struct f2fs_cached_block *__get_meta_cache(struct f2fs_sb_info *sbi, + pgoff_t index, bool is_meta) { - struct address_space *mapping = META_MAPPING(sbi); - struct folio *folio; + struct f2fs_cached_block *entry; struct f2fs_io_info fio = { .sbi = sbi, .type = META, @@ -265,70 +266,75 @@ static struct folio *__get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index, .new_blkaddr = index, .encrypted_page = NULL, .is_por = !is_meta ? 1 : 0, + .is_cache = 1, }; int err; if (unlikely(!is_meta)) fio.op_flags &= ~REQ_META; repeat: - folio = f2fs_grab_cache_folio(mapping, index, false); - if (IS_ERR(folio)) { + entry = f2fs_grab_cache(META_CACHE(sbi), index, + F2FS_CACHE_LOCK_CREATE); + if (IS_ERR(entry)) { cond_resched(); goto repeat; } - if (folio_test_uptodate(folio)) + if (f2fs_cache_test_uptodate(entry)) goto out; - fio.folio = folio; + fio.cache_entry = entry; - err = f2fs_submit_page_bio(&fio); + err = f2fs_submit_cache_read(&fio); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); return ERR_PTR(err); } f2fs_update_iostat(sbi, NULL, FS_META_READ_IO, F2FS_BLKSIZE); - folio_lock(folio); - if (unlikely(!is_meta_folio(folio))) { - f2fs_folio_put(folio, true); + f2fs_lock_cache(entry); + if (unlikely(!f2fs_is_meta_cache(entry))) { + f2fs_put_cache(entry, true); goto repeat; } - if (unlikely(!folio_test_uptodate(folio))) { - f2fs_handle_page_eio(sbi, folio, META); - f2fs_folio_put(folio, true); + if (unlikely(!f2fs_cache_test_uptodate(entry))) { + f2fs_handle_page_eio(sbi, entry->index, META); + f2fs_put_cache(entry, true); return ERR_PTR(-EIO); } out: - return folio; + return entry; } -struct folio *f2fs_get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index) +struct f2fs_cached_block *f2fs_get_meta_cache(struct f2fs_sb_info *sbi, + pgoff_t index) { - return __get_meta_folio(sbi, index, true); + return __get_meta_cache(sbi, index, true); } -struct folio *f2fs_get_meta_folio_retry(struct f2fs_sb_info *sbi, pgoff_t index) +struct f2fs_cached_block *f2fs_get_meta_cache_retry(struct f2fs_sb_info *sbi, + pgoff_t index) { - struct folio *folio; + struct f2fs_cached_block *entry; int count = 0; retry: - folio = __get_meta_folio(sbi, index, true); - if (IS_ERR(folio)) { - if (PTR_ERR(folio) == -EIO && + entry = __get_meta_cache(sbi, index, true); + if (IS_ERR(entry)) { + if (PTR_ERR(entry) == -EIO && ++count <= DEFAULT_RETRY_IO_COUNT) goto retry; f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_META_PAGE); } - return folio; + return entry; } /* for POR only */ -struct folio *f2fs_get_tmp_folio(struct f2fs_sb_info *sbi, pgoff_t index) +struct f2fs_cached_block *f2fs_get_tmp_cache(struct f2fs_sb_info *sbi, + pgoff_t index) { - return __get_meta_folio(sbi, index, false); + return __get_meta_cache(sbi, index, false); } static bool __is_bitmap_valid(struct f2fs_sb_info *sbi, block_t blkaddr, @@ -446,9 +452,10 @@ bool f2fs_is_valid_blkaddr_raw(struct f2fs_sb_info *sbi, /* * Readahead CP/NAT/SIT/SSA/POR pages */ -int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, - int type, bool sync) +int f2fs_ra_meta_caches(struct f2fs_sb_info *sbi, block_t start, + int nrpages, int type, bool sync) { + struct f2fs_cached_block_list *cache = META_CACHE(sbi); block_t blkno = start; struct f2fs_io_info fio = { .sbi = sbi, @@ -458,6 +465,7 @@ int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, .encrypted_page = NULL, .in_list = 0, .is_por = (type == META_POR) ? 1 : 0, + .is_cache = 1, }; struct blk_plug plug; int err; @@ -467,7 +475,7 @@ int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, blk_start_plug(&plug); for (; nrpages-- > 0; blkno++) { - struct folio *folio; + struct f2fs_cached_block *entry; if (!f2fs_is_valid_blkaddr(sbi, blkno, type)) goto out; @@ -494,62 +502,58 @@ int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, fio.new_blkaddr = blkno; break; default: - BUG(); + f2fs_bug_on(sbi, 1); } - folio = f2fs_grab_cache_folio(META_MAPPING(sbi), - fio.new_blkaddr, false); - if (IS_ERR(folio)) + entry = f2fs_grab_cache(cache, fio.new_blkaddr, + F2FS_CACHE_LOCK_CREATE); + if (IS_ERR(entry)) continue; - if (folio_test_uptodate(folio)) { - f2fs_folio_put(folio, true); + if (f2fs_cache_test_uptodate(entry)) { + f2fs_put_cache(entry, true); continue; } - fio.folio = folio; - err = f2fs_submit_page_bio(&fio); - f2fs_folio_put(folio, err ? true : false); + fio.cache_entry = entry; + err = f2fs_submit_cache_read(&fio); + f2fs_put_cache(entry, err ? true : false); if (!err) f2fs_update_iostat(sbi, NULL, FS_META_READ_IO, - F2FS_BLKSIZE); + sbi->blocksize); } out: blk_finish_plug(&plug); return blkno - start; } -void f2fs_ra_meta_pages_cond(struct f2fs_sb_info *sbi, pgoff_t index, - unsigned int ra_blocks) +void f2fs_ra_meta_caches_cond(struct f2fs_sb_info *sbi, pgoff_t index, + unsigned int ra_blocks) { - struct folio *folio; + struct f2fs_cached_block *entry; bool readahead = false; if (ra_blocks == RECOVERY_MIN_RA_BLOCKS) return; - folio = filemap_get_folio(META_MAPPING(sbi), index); - if (IS_ERR(folio) || !folio_test_uptodate(folio)) + entry = f2fs_find_cache(META_CACHE(sbi), index); + if (IS_ERR(entry) || !f2fs_cache_test_uptodate(entry)) readahead = true; - f2fs_folio_put(folio, false); + f2fs_put_cache(entry, false); if (readahead) - f2fs_ra_meta_pages(sbi, index, ra_blocks, META_POR, true); + f2fs_ra_meta_caches(sbi, index, ra_blocks, META_POR, true); } -static bool __f2fs_write_meta_folio(struct folio *folio, - struct writeback_control *wbc, +static bool __f2fs_write_meta_cache(struct f2fs_cached_block *entry, enum iostat_type io_type) { - struct f2fs_sb_info *sbi = F2FS_F_SB(folio); - - trace_f2fs_writepage(folio, META); + struct f2fs_sb_info *sbi = entry->cache->sbi; if (unlikely(f2fs_cp_error(sbi))) { if (is_sbi_flag_set(sbi, SBI_IS_CLOSE)) { - folio_clear_uptodate(folio); - dec_page_count(sbi, F2FS_DIRTY_META); - folio_unlock(folio); + f2fs_force_clear_cache_dirty(entry); + f2fs_unlock_cache(entry); return true; } goto redirty_out; @@ -557,10 +561,10 @@ static bool __f2fs_write_meta_folio(struct folio *folio, if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING))) goto redirty_out; - f2fs_do_write_meta_page(sbi, folio, io_type); + f2fs_do_write_meta_cache(sbi, entry, io_type); dec_page_count(sbi, F2FS_DIRTY_META); - folio_unlock(folio); + f2fs_unlock_cache(entry); if (unlikely(f2fs_cp_error(sbi))) f2fs_submit_merged_write(sbi, META); @@ -568,101 +572,89 @@ static bool __f2fs_write_meta_folio(struct folio *folio, return true; redirty_out: - folio_redirty_for_writepage(wbc, folio); + f2fs_cache_set_dirty(entry); return false; } -static int f2fs_write_meta_pages(struct address_space *mapping, - struct writeback_control *wbc) +void f2fs_write_meta_caches(struct f2fs_sb_info *sbi) { - struct f2fs_sb_info *sbi = F2FS_M_SB(mapping); struct f2fs_lock_context lc; - long diff, written; + long nr_to_write = LONG_MAX; if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING))) - goto skip_write; + return; - /* collect a number of dirty meta pages and write together */ - if (wbc->sync_mode != WB_SYNC_ALL && - get_pages(sbi, F2FS_DIRTY_META) < - nr_pages_to_skip(sbi, META)) - goto skip_write; + /* collect a number of dirty meta caches and write together */ + if (get_pages(sbi, F2FS_DIRTY_META) < + nr_pages_to_skip(sbi, META)) + return; - /* if locked failed, cp will flush dirty pages instead */ + /* if locked failed, cp will flush dirty caches instead */ if (!f2fs_down_write_trylock_trace(&sbi->cp_global_sem, &lc)) - goto skip_write; + return; - trace_f2fs_writepages(mapping->host, wbc, META); - diff = nr_pages_to_write(sbi, META, wbc); - written = f2fs_sync_meta_pages(sbi, wbc->nr_to_write, FS_META_IO); + nr_to_write = adjust_flush_cache_number(sbi, META); + f2fs_sync_meta_caches(sbi, nr_to_write, false, FS_META_IO); f2fs_up_write_trace(&sbi->cp_global_sem, &lc); - wbc->nr_to_write = max((long)0, wbc->nr_to_write - written - diff); - return 0; - -skip_write: - wbc->pages_skipped += get_pages(sbi, F2FS_DIRTY_META); - trace_f2fs_writepages(mapping->host, wbc, META); - return 0; } -long f2fs_sync_meta_pages(struct f2fs_sb_info *sbi, long nr_to_write, - enum iostat_type io_type) +long f2fs_sync_meta_caches(struct f2fs_sb_info *sbi, long nr_to_write, + bool sync, enum iostat_type io_type) { - struct address_space *mapping = META_MAPPING(sbi); pgoff_t index = 0, prev = ULONG_MAX; - struct folio_batch fbatch; + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; long nwritten = 0; - int nr_folios; - struct writeback_control wbc = {}; + int nr; struct blk_plug plug; - - folio_batch_init(&fbatch); + bool background = nr_to_write != LONG_MAX; blk_start_plug(&plug); - while ((nr_folios = filemap_get_folios_tag(mapping, &index, - (pgoff_t)-1, - PAGECACHE_TAG_DIRTY, &fbatch))) { + while ((nr = f2fs_cache_gang_lookup_tag(META_CACHE(sbi), entries, + &index, F2FS_ONSTACK_CACHES, F2FS_CACHE_TAG_DIRTY))) { int i; - for (i = 0; i < nr_folios; i++) { - struct folio *folio = fbatch.folios[i]; + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; + + if (background && unlikely(freezing(current))) { + f2fs_cache_gang_release(entries, nr); + goto stop; + } - if (nr_to_write != LONG_MAX && i != 0 && - folio->index != prev + - folio_nr_pages(fbatch.folios[i-1])) { - folio_batch_release(&fbatch); + if (background && i != 0 && + entry->index != prev + 1) { + f2fs_cache_gang_release(entries, nr); goto stop; } - folio_lock(folio); + f2fs_lock_cache(entry); - if (unlikely(!is_meta_folio(folio))) { + if (unlikely(!f2fs_is_meta_cache(entry))) { continue_unlock: - folio_unlock(folio); + f2fs_unlock_cache(entry); continue; } - if (!folio_test_dirty(folio)) { + if (!f2fs_cache_test_dirty(entry)) { /* someone wrote it for us */ goto continue_unlock; } - f2fs_folio_wait_writeback(folio, META, true, true); + f2fs_cache_wait_writeback(entry); - if (!folio_clear_dirty_for_io(folio)) + if (!f2fs_clear_cache_dirty(entry)) goto continue_unlock; - if (!__f2fs_write_meta_folio(folio, &wbc, - io_type)) { - folio_unlock(folio); + if (!__f2fs_write_meta_cache(entry, io_type)) { + f2fs_unlock_cache(entry); break; } - nwritten += folio_nr_pages(folio); - prev = folio->index; + nwritten += sbi->blocksize; + prev = entry->index; if (unlikely(nwritten >= nr_to_write)) break; } - folio_batch_release(&fbatch); + f2fs_cache_gang_release(entries, nr); cond_resched(); } stop: @@ -674,29 +666,6 @@ long f2fs_sync_meta_pages(struct f2fs_sb_info *sbi, long nr_to_write, return nwritten; } -static bool f2fs_dirty_meta_folio(struct address_space *mapping, - struct folio *folio) -{ - trace_f2fs_set_page_dirty(folio, META); - - if (!folio_test_uptodate(folio)) - folio_mark_uptodate(folio); - if (filemap_dirty_folio(mapping, folio)) { - inc_page_count(F2FS_M_SB(mapping), F2FS_DIRTY_META); - folio_set_f2fs_reference(folio); - return true; - } - return false; -} - -const struct address_space_operations f2fs_meta_aops = { - .writepages = f2fs_write_meta_pages, - .dirty_folio = f2fs_dirty_meta_folio, - .invalidate_folio = f2fs_invalidate_folio, - .release_folio = f2fs_release_folio, - .migrate_folio = filemap_migrate_folio, -}; - static void __add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, unsigned int devidx, int type) { @@ -1036,20 +1005,20 @@ int f2fs_recover_orphan_inodes(struct f2fs_sb_info *sbi) start_blk = __start_cp_addr(sbi) + 1 + __cp_payload(sbi); orphan_blocks = __start_sum_addr(sbi) - 1 - __cp_payload(sbi); - f2fs_ra_meta_pages(sbi, start_blk, orphan_blocks, META_CP, true); + f2fs_ra_meta_caches(sbi, start_blk, orphan_blocks, META_CP, true); for (i = 0; i < orphan_blocks; i++) { - struct folio *folio; + struct f2fs_cached_block *entry; struct f2fs_orphan_block *orphan_blk; unsigned int entry_count; - folio = f2fs_get_meta_folio(sbi, start_blk + i); - if (IS_ERR(folio)) { - err = PTR_ERR(folio); + entry = f2fs_get_meta_cache(sbi, start_blk + i); + if (IS_ERR(entry)) { + err = PTR_ERR(entry); goto out; } - orphan_blk = folio_address(folio); + orphan_blk = cache_address(entry); entry_count = le32_to_cpu(orphan_blk->entry_count); if (entry_count > F2FS_ORPHANS_PER_BLOCK) { f2fs_err(sbi, "invalid orphan inode entry count %u", @@ -1057,7 +1026,7 @@ int f2fs_recover_orphan_inodes(struct f2fs_sb_info *sbi) set_sbi_flag(sbi, SBI_NEED_FSCK); f2fs_handle_error(sbi, ERROR_INCONSISTENT_ORPHAN); err = -EFSCORRUPTED; - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); goto out; } @@ -1066,11 +1035,11 @@ int f2fs_recover_orphan_inodes(struct f2fs_sb_info *sbi) err = recover_orphan_inode(sbi, ino); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); goto out; } } - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); } /* clear Orphan Flag */ clear_ckpt_flags(sbi, CP_ORPHAN_PRESENT_FLAG); @@ -1087,9 +1056,9 @@ static void write_orphan_inodes(struct f2fs_sb_info *sbi, block_t start_blk) unsigned int nentries = 0; unsigned short index = 1; unsigned short orphan_blocks; - struct folio *folio = NULL; struct ino_entry *orphan = NULL; struct inode_management *im = &sbi->im[ORPHAN_INO]; + struct f2fs_cached_block *entry = NULL; orphan_blocks = GET_ORPHAN_BLOCKS(im->ino_num); @@ -1102,9 +1071,9 @@ static void write_orphan_inodes(struct f2fs_sb_info *sbi, block_t start_blk) /* loop for each orphan inode entry and write them in journal block */ list_for_each_entry(orphan, head, list) { - if (!folio) { - folio = f2fs_grab_meta_folio(sbi, start_blk++); - orphan_blk = folio_address(folio); + if (!entry) { + entry = f2fs_grab_meta_cache(sbi, start_blk++); + orphan_blk = (struct f2fs_orphan_block *)cache_address(entry); memset(orphan_blk, 0, sizeof(*orphan_blk)); } @@ -1119,20 +1088,20 @@ static void write_orphan_inodes(struct f2fs_sb_info *sbi, block_t start_blk) orphan_blk->blk_addr = cpu_to_le16(index); orphan_blk->blk_count = cpu_to_le16(orphan_blocks); orphan_blk->entry_count = cpu_to_le32(nentries); - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); + f2fs_mark_cache_dirty(entry); + f2fs_put_cache(entry, true); index++; nentries = 0; - folio = NULL; + entry = NULL; } } - if (folio) { + if (entry) { orphan_blk->blk_addr = cpu_to_le16(index); orphan_blk->blk_count = cpu_to_le16(orphan_blocks); orphan_blk->entry_count = cpu_to_le32(nentries); - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); + f2fs_mark_cache_dirty(entry); + f2fs_put_cache(entry, true); } } @@ -1151,29 +1120,29 @@ static __u32 f2fs_checkpoint_chksum(struct f2fs_checkpoint *ckpt) } static int get_checkpoint_version(struct f2fs_sb_info *sbi, block_t cp_addr, - struct f2fs_checkpoint **cp_block, struct folio **cp_folio, + struct f2fs_checkpoint **cp_block, struct f2fs_cached_block **cp_entry, unsigned long long *version) { size_t crc_offset = 0; __u32 crc; - *cp_folio = f2fs_get_meta_folio(sbi, cp_addr); - if (IS_ERR(*cp_folio)) - return PTR_ERR(*cp_folio); + *cp_entry = f2fs_get_meta_cache(sbi, cp_addr); + if (IS_ERR(*cp_entry)) + return PTR_ERR(*cp_entry); - *cp_block = folio_address(*cp_folio); + *cp_block = (struct f2fs_checkpoint *)cache_address(*cp_entry); crc_offset = le32_to_cpu((*cp_block)->checksum_offset); if (crc_offset < CP_MIN_CHKSUM_OFFSET || crc_offset > CP_CHKSUM_OFFSET) { - f2fs_folio_put(*cp_folio, true); + f2fs_put_cache(*cp_entry, true); f2fs_warn(sbi, "invalid crc_offset: %zu", crc_offset); return -EINVAL; } crc = f2fs_checkpoint_chksum(*cp_block); if (crc != cur_cp_crc(*cp_block)) { - f2fs_folio_put(*cp_folio, true); + f2fs_put_cache(*cp_entry, true); f2fs_warn(sbi, "invalid crc value"); return -EINVAL; } @@ -1182,17 +1151,17 @@ static int get_checkpoint_version(struct f2fs_sb_info *sbi, block_t cp_addr, return 0; } -static struct folio *validate_checkpoint(struct f2fs_sb_info *sbi, +static struct f2fs_cached_block *validate_checkpoint(struct f2fs_sb_info *sbi, block_t cp_addr, unsigned long long *version) { - struct folio *cp_folio_1 = NULL, *cp_folio_2 = NULL; + struct f2fs_cached_block *cp_entry_1 = NULL, *cp_entry_2 = NULL; struct f2fs_checkpoint *cp_block = NULL; unsigned long long cur_version = 0, pre_version = 0; unsigned int cp_blocks; int err; err = get_checkpoint_version(sbi, cp_addr, &cp_block, - &cp_folio_1, version); + &cp_entry_1, version); if (err) return NULL; @@ -1207,19 +1176,20 @@ static struct folio *validate_checkpoint(struct f2fs_sb_info *sbi, cp_addr += cp_blocks - 1; err = get_checkpoint_version(sbi, cp_addr, &cp_block, - &cp_folio_2, version); + &cp_entry_2, version); if (err) goto invalid_cp; cur_version = *version; if (cur_version == pre_version) { *version = cur_version; - f2fs_folio_put(cp_folio_2, true); - return cp_folio_1; + f2fs_put_cache(cp_entry_2, true); + return cp_entry_1; } - f2fs_folio_put(cp_folio_2, true); + f2fs_put_cache(cp_entry_2, true); invalid_cp: - f2fs_folio_put(cp_folio_1, true); + if (cp_entry_1) + f2fs_put_cache(cp_entry_1, true); return NULL; } @@ -1227,7 +1197,7 @@ int f2fs_get_valid_checkpoint(struct f2fs_sb_info *sbi) { struct f2fs_checkpoint *cp_block; struct f2fs_super_block *fsb = sbi->raw_super; - struct folio *cp1, *cp2, *cur_folio; + struct f2fs_cached_block *cp1 = NULL, *cp2 = NULL, *cur_entry; unsigned long blk_size = sbi->blocksize; unsigned long long cp1_version = 0, cp2_version = 0; unsigned long long cp_start_blk_no; @@ -1254,22 +1224,22 @@ int f2fs_get_valid_checkpoint(struct f2fs_sb_info *sbi) if (cp1 && cp2) { if (ver_after(cp2_version, cp1_version)) - cur_folio = cp2; + cur_entry = cp2; else - cur_folio = cp1; + cur_entry = cp1; } else if (cp1) { - cur_folio = cp1; + cur_entry = cp1; } else if (cp2) { - cur_folio = cp2; + cur_entry = cp2; } else { err = -EFSCORRUPTED; goto fail_no_cp; } - cp_block = folio_address(cur_folio); + cp_block = (struct f2fs_checkpoint *)cache_address(cur_entry); memcpy(sbi->ckpt, cp_block, blk_size); - if (cur_folio == cp1) + if (cur_entry == cp1) sbi->cur_cp_pack = 1; else sbi->cur_cp_pack = 2; @@ -1284,30 +1254,35 @@ int f2fs_get_valid_checkpoint(struct f2fs_sb_info *sbi) goto done; cp_blk_no = le32_to_cpu(fsb->cp_blkaddr); - if (cur_folio == cp2) + if (cur_entry == cp2) cp_blk_no += BIT(le32_to_cpu(fsb->log_blocks_per_seg)); for (i = 1; i < cp_blks; i++) { + struct f2fs_cached_block *cur_entry_payload; void *sit_bitmap_ptr; unsigned char *ckpt = (unsigned char *)sbi->ckpt; - cur_folio = f2fs_get_meta_folio(sbi, cp_blk_no + i); - if (IS_ERR(cur_folio)) { - err = PTR_ERR(cur_folio); + cur_entry_payload = f2fs_get_meta_cache(sbi, cp_blk_no + i); + if (IS_ERR(cur_entry_payload)) { + err = PTR_ERR(cur_entry_payload); goto free_fail_no_cp; } - sit_bitmap_ptr = folio_address(cur_folio); + sit_bitmap_ptr = cache_address(cur_entry_payload); memcpy(ckpt + i * blk_size, sit_bitmap_ptr, blk_size); - f2fs_folio_put(cur_folio, true); + f2fs_put_cache(cur_entry_payload, true); } done: - f2fs_folio_put(cp1, true); - f2fs_folio_put(cp2, true); + if (cp1) + f2fs_put_cache(cp1, true); + if (cp2) + f2fs_put_cache(cp2, true); return 0; free_fail_no_cp: - f2fs_folio_put(cp1, true); - f2fs_folio_put(cp2, true); + if (cp1) + f2fs_put_cache(cp1, true); + if (cp2) + f2fs_put_cache(cp2, true); fail_no_cp: kvfree(sbi->ckpt); return err; @@ -1621,7 +1596,7 @@ void f2fs_wait_on_all_pages(struct f2fs_sb_info *sbi, int type) break; if (type == F2FS_DIRTY_META) - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_CP_META_IO); + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_META_IO); else if (type == F2FS_WB_CP_DATA) f2fs_submit_merged_write(sbi, DATA); @@ -1700,31 +1675,24 @@ static void update_ckpt_flags(struct f2fs_sb_info *sbi, struct cp_control *cpc) static void commit_checkpoint(struct f2fs_sb_info *sbi, void *src, block_t blk_addr) { - struct writeback_control wbc = {}; - - /* - * filemap_get_folios_tag and folio_lock again will take - * some extra time. Therefore, f2fs_update_meta_pages and - * f2fs_sync_meta_pages are combined in this function. - */ - struct folio *folio = f2fs_grab_meta_folio(sbi, blk_addr); + struct f2fs_cached_block *entry = f2fs_grab_meta_cache(sbi, blk_addr); - memcpy(folio_address(folio), src, PAGE_SIZE); + memcpy(cache_address(entry), src, F2FS_BLKSIZE); - folio_mark_dirty(folio); - if (unlikely(!folio_clear_dirty_for_io(folio))) + f2fs_mark_cache_dirty(entry); + if (unlikely(!f2fs_clear_cache_dirty(entry))) f2fs_bug_on(sbi, 1); /* writeout cp pack 2 page */ - if (unlikely(!__f2fs_write_meta_folio(folio, &wbc, FS_CP_META_IO))) { + if (unlikely(!__f2fs_write_meta_cache(entry, FS_CP_META_IO))) { if (f2fs_cp_error(sbi)) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); return; } f2fs_bug_on(sbi, true); } - f2fs_folio_put(folio, false); + f2fs_put_cache(entry, false); /* submit checkpoint (with barrier if NOBARRIER is not set) */ f2fs_submit_merged_write(sbi, META_FLUSH); @@ -1793,7 +1761,7 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc) int err; /* Flush all the NAT/SIT pages */ - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_CP_META_IO); + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_CP_META_IO); stat_cp_time(cpc, CP_TIME_SYNC_META); @@ -1892,7 +1860,7 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc) } /* Here, we have one bio having CP pack except cp pack 2 page */ - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_CP_META_IO); + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_CP_META_IO); stat_cp_time(cpc, CP_TIME_SYNC_CP_META); /* Wait for all dirty meta pages to be submitted for IO */ @@ -1919,10 +1887,10 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc) * used for migration of encrypted, verity or compressed inode's blocks. */ if (f2fs_sb_has_encrypt(sbi) || f2fs_sb_has_verity(sbi) || - f2fs_sb_has_compression(sbi)) - f2fs_bug_on(sbi, - invalidate_inode_pages2_range(META_MAPPING(sbi), - MAIN_BLKADDR(sbi), MAX_BLKADDR(sbi) - 1)); + f2fs_sb_has_compression(sbi)) { + f2fs_truncate_meta_caches(sbi, MAIN_BLKADDR(sbi), + MAX_BLKADDR(sbi) - MAIN_BLKADDR(sbi)); + } f2fs_release_ino_entry(sbi, false); diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c index 91855d91bbdd..bb749f6257a1 100644 --- a/fs/f2fs/compress.c +++ b/fs/f2fs/compress.c @@ -1150,7 +1150,7 @@ static int prepare_compress_overwrite(struct compress_ctx *cc, f2fs_compress_ctx_add_page(cc, folio); if (!folio_test_uptodate(folio)) { - f2fs_handle_page_eio(sbi, folio, DATA); + f2fs_handle_page_eio(sbi, folio->index, DATA); release_and_retry: f2fs_put_rpages(cc); f2fs_unlock_rpages(cc, i + 1); @@ -1359,7 +1359,7 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc, fio.old_blkaddr = data_blkaddr(dn.inode, dn.node_folio, dn.ofs_in_node + i + 1); - /* wait for GCed page writeback via META_MAPPING */ + /* wait for GCed page writeback via generic cache */ f2fs_wait_on_block_writeback(inode, fio.old_blkaddr); if (fio.encrypted) { diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index 110282bb8dcd..92c3293f0a1e 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -65,8 +65,7 @@ bool f2fs_is_cp_guaranteed(const struct folio *folio) inode = mapping->host; - if (inode->i_ino == F2FS_META_INO(sbi) || - inode->i_ino == F2FS_NODE_INO(sbi) || + if (inode->i_ino == F2FS_NODE_INO(sbi) || S_ISDIR(inode->i_mode)) return true; @@ -84,9 +83,6 @@ static enum count_type __read_io_type(struct folio *folio) struct inode *inode = mapping->host; struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - if (inode->i_ino == F2FS_META_INO(sbi)) - return F2FS_RD_META; - if (inode->i_ino == F2FS_NODE_INO(sbi)) return F2FS_RD_NODE; } @@ -1452,7 +1448,7 @@ static void f2fs_submit_page_read(struct inode *inode, struct fsverity_info *vi, bio = f2fs_grab_read_bio(inode, vi, blkaddr, 1, op_flags, folio->index, for_write); - /* wait for GCed page writeback via META_MAPPING */ + /* wait for GCed page writeback via generic cache */ f2fs_wait_on_block_writeback(inode, blkaddr); if (!bio_add_folio(bio, folio, PAGE_SIZE, 0)) @@ -3110,7 +3106,7 @@ static void f2fs_readahead(struct readahead_control *rac) int f2fs_encrypt_one_page(struct f2fs_io_info *fio) { struct inode *inode = fio_inode(fio); - struct folio *mfolio; + struct f2fs_cached_block *entry; struct page *page; if (!f2fs_encrypted_file(inode)) @@ -3126,12 +3122,13 @@ int f2fs_encrypt_one_page(struct f2fs_io_info *fio) if (IS_ERR(fio->encrypted_page)) return PTR_ERR(fio->encrypted_page); - mfolio = filemap_lock_folio(META_MAPPING(fio->sbi), fio->old_blkaddr); - if (!IS_ERR(mfolio)) { - if (folio_test_uptodate(mfolio)) - memcpy(folio_address(mfolio), - page_address(fio->encrypted_page), PAGE_SIZE); - f2fs_folio_put(mfolio, true); + entry = f2fs_find_cache(META_CACHE(fio->sbi), fio->old_blkaddr); + if (!IS_ERR(entry)) { + f2fs_lock_cache(entry); + if (f2fs_cache_test_uptodate(entry)) + memcpy(cache_address(entry), page_address(fio->encrypted_page), + F2FS_BLKSIZE); + f2fs_put_cache(entry, true); } return 0; } @@ -3297,7 +3294,7 @@ int f2fs_do_write_data_page(struct f2fs_io_info *fio) goto out_writepage; } - /* wait for GCed page writeback via META_MAPPING */ + /* wait for GCed page writeback via generic cache */ if (fio->meta_gc) f2fs_wait_on_block_writeback(inode, fio->old_blkaddr); @@ -4380,9 +4377,7 @@ void f2fs_invalidate_folio(struct folio *folio, size_t offset, size_t length) return; if (folio_test_dirty(folio)) { - if (inode->i_ino == F2FS_META_INO(sbi)) { - dec_page_count(sbi, F2FS_DIRTY_META); - } else if (inode->i_ino == F2FS_NODE_INO(sbi)) { + if (inode->i_ino == F2FS_NODE_INO(sbi)) { dec_page_count(sbi, F2FS_DIRTY_NODES); } else { inode_dec_dirty_pages(inode); diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c index ff379aff4472..14059a50739c 100644 --- a/fs/f2fs/debug.c +++ b/fs/f2fs/debug.c @@ -224,8 +224,7 @@ static void update_general_status(struct f2fs_sb_info *sbi) si->dirty_count = dirty_segments(sbi); if (sbi->node_inode) si->node_pages = NODE_MAPPING(sbi)->nrpages; - if (sbi->meta_inode) - si->meta_pages = META_MAPPING(sbi)->nrpages; + si->meta_caches = META_CACHE(sbi)->num_entries; #ifdef CONFIG_F2FS_FS_COMPRESSION if (sbi->compress_inode) { si->compress_pages = COMPRESS_MAPPING(sbi)->nrpages; @@ -388,11 +387,8 @@ static void update_mem_info(struct f2fs_sb_info *sbi) si->page_mem += (unsigned long long)npages << PAGE_SHIFT; } - if (sbi->meta_inode) { - unsigned long npages = META_MAPPING(sbi)->nrpages; - - si->page_mem += (unsigned long long)npages << PAGE_SHIFT; - } + si->page_mem += (unsigned long long)META_CACHE(sbi)->num_entries << PAGE_SHIFT; + si->cache_mem += META_CACHE(sbi)->num_entries * sizeof(struct f2fs_cached_block); #ifdef CONFIG_F2FS_FS_COMPRESSION if (sbi->compress_inode) { unsigned long npages = COMPRESS_MAPPING(sbi)->nrpages; @@ -708,7 +704,7 @@ static int stat_show(struct seq_file *s, void *v) seq_printf(s, " - quota data: %4d in quota files:%4d\n", si->ndirty_qdata, si->nquota_files); seq_printf(s, " - meta: %4d in %4d\n", - si->ndirty_meta, si->meta_pages); + si->ndirty_meta, si->meta_caches); seq_printf(s, " - imeta: %4d\n", si->ndirty_imeta); seq_printf(s, " - fsync mark: %4lld\n", diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 6e20b3586f26..9a353dcd6658 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -1828,7 +1828,6 @@ struct f2fs_sb_info { struct f2fs_checkpoint *ckpt; /* raw checkpoint pointer */ int cur_cp_pack; /* remain current cp pack */ spinlock_t cp_lock; /* for flag in ckpt */ - struct inode *meta_inode; /* cache meta blocks */ struct f2fs_rwsem cp_global_sem; /* checkpoint procedure lock */ struct f2fs_rwsem cp_rwsem; /* blocking FS operations */ struct f2fs_rwsem node_write; /* locking node writes */ @@ -1873,7 +1872,6 @@ struct f2fs_sb_info { unsigned int blocksize; /* block size */ unsigned int root_ino_num; /* root inode number*/ unsigned int node_ino_num; /* node inode number*/ - unsigned int meta_ino_num; /* meta inode number*/ unsigned int log_blocks_per_seg; /* log2 blocks per segment */ unsigned int blocks_per_seg; /* blocks per segment */ unsigned int segs_per_sec; /* segments per section */ @@ -2319,9 +2317,9 @@ static inline struct dirty_seglist_info *DIRTY_I(struct f2fs_sb_info *sbi) return (struct dirty_seglist_info *)(SM_I(sbi)->dirty_info); } -static inline struct address_space *META_MAPPING(struct f2fs_sb_info *sbi) +static inline bool f2fs_is_meta_cache(struct f2fs_cached_block *entry) { - return sbi->meta_inode->i_mapping; + return entry->cache && entry->cache == META_CACHE(entry->cache->sbi); } static inline struct address_space *NODE_MAPPING(struct f2fs_sb_info *sbi) @@ -2329,11 +2327,6 @@ static inline struct address_space *NODE_MAPPING(struct f2fs_sb_info *sbi) return sbi->node_inode->i_mapping; } -static inline bool is_meta_folio(struct folio *folio) -{ - return folio->mapping == META_MAPPING(F2FS_F_SB(folio)); -} - static inline bool is_node_folio(struct folio *folio) { return folio->mapping == NODE_MAPPING(F2FS_F_SB(folio)); @@ -4057,9 +4050,9 @@ bool f2fs_alloc_nid(struct f2fs_sb_info *sbi, nid_t *nid); void f2fs_alloc_nid_done(struct f2fs_sb_info *sbi, nid_t nid); void f2fs_alloc_nid_failed(struct f2fs_sb_info *sbi, nid_t nid); int f2fs_try_to_free_nids(struct f2fs_sb_info *sbi, int nr_shrink); -int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio); -int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio); -int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct folio *folio); +int f2fs_recover_inline_xattr(struct inode *inode, struct f2fs_cached_block *entry); +int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry); +int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry); int f2fs_restore_node_summary(struct f2fs_sb_info *sbi, unsigned int segno, struct f2fs_summary_block *sum); int f2fs_flush_nat_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc); @@ -4109,11 +4102,13 @@ int f2fs_allocate_new_segments(struct f2fs_sb_info *sbi); int f2fs_trim_fs(struct f2fs_sb_info *sbi, struct fstrim_range *range); bool f2fs_exist_trim_candidates(struct f2fs_sb_info *sbi, struct cp_control *cpc); -struct folio *f2fs_get_sum_folio(struct f2fs_sb_info *sbi, unsigned int segno); +struct f2fs_cached_block *f2fs_get_sum_cache(struct f2fs_sb_info *sbi, + unsigned int segno); void f2fs_update_meta_page(struct f2fs_sb_info *sbi, void *src, block_t blk_addr); -void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct folio *folio, - enum iostat_type io_type); +void f2fs_do_write_meta_cache(struct f2fs_sb_info *sbi, + struct f2fs_cached_block *entry, + enum iostat_type io_type); void f2fs_do_write_node_page(unsigned int nid, struct f2fs_io_info *fio); void f2fs_outplace_write_data(struct dnode_of_data *dn, struct f2fs_io_info *fio); @@ -4136,8 +4131,6 @@ void f2fs_update_device_state(struct f2fs_sb_info *sbi, nid_t ino, block_t blkaddr, unsigned int blkcnt); void f2fs_folio_wait_writeback(struct folio *folio, enum page_type type, bool ordered, bool locked); -#define f2fs_wait_on_page_writeback(page, type, ordered, locked) \ - f2fs_folio_wait_writeback(page_folio(page), type, ordered, locked) void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr); void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr, block_t len); @@ -4203,20 +4196,21 @@ void f2fs_unlock_op(struct f2fs_sb_info *sbi, struct f2fs_lock_context *lc); void f2fs_stop_checkpoint(struct f2fs_sb_info *sbi, bool end_io, unsigned char reason); void f2fs_flush_ckpt_thread(struct f2fs_sb_info *sbi); -struct folio *f2fs_grab_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index); -struct folio *f2fs_get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index); -struct folio *f2fs_get_meta_folio_retry(struct f2fs_sb_info *sbi, pgoff_t index); -struct folio *f2fs_get_tmp_folio(struct f2fs_sb_info *sbi, pgoff_t index); +struct f2fs_cached_block *f2fs_grab_meta_cache(struct f2fs_sb_info *sbi, pgoff_t index); +struct f2fs_cached_block *f2fs_get_meta_cache(struct f2fs_sb_info *sbi, pgoff_t index); +struct f2fs_cached_block *f2fs_get_meta_cache_retry(struct f2fs_sb_info *sbi, pgoff_t index); +struct f2fs_cached_block *f2fs_get_tmp_cache(struct f2fs_sb_info *sbi, pgoff_t index); bool f2fs_is_valid_blkaddr(struct f2fs_sb_info *sbi, block_t blkaddr, int type); bool f2fs_is_valid_blkaddr_raw(struct f2fs_sb_info *sbi, block_t blkaddr, int type); -int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, +int f2fs_ra_meta_caches(struct f2fs_sb_info *sbi, block_t start, int nrpages, int type, bool sync); -void f2fs_ra_meta_pages_cond(struct f2fs_sb_info *sbi, pgoff_t index, +void f2fs_ra_meta_caches_cond(struct f2fs_sb_info *sbi, pgoff_t index, unsigned int ra_blocks); -long f2fs_sync_meta_pages(struct f2fs_sb_info *sbi, long nr_to_write, - enum iostat_type io_type); +void f2fs_write_meta_caches(struct f2fs_sb_info *sbi); +long f2fs_sync_meta_caches(struct f2fs_sb_info *sbi, long nr_to_write, + bool sync, enum iostat_type io_type); void f2fs_add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type); void f2fs_remove_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type); void f2fs_release_ino_entry(struct f2fs_sb_info *sbi, bool all); @@ -4403,7 +4397,7 @@ struct f2fs_stat_info { unsigned int bimodal, avg_vblocks; int util_free, util_valid, util_invalid; int rsvd_segs, overp_segs; - int dirty_count, node_pages, meta_pages, compress_pages; + int dirty_count, node_pages, meta_caches, compress_pages; int compress_page_hit; int prefree_count, free_segs, free_secs; int cp_call_count[MAX_CALL_TYPE], cp_count; @@ -4605,7 +4599,6 @@ extern const struct file_operations f2fs_file_operations; extern const struct inode_operations f2fs_file_inode_operations; extern const struct address_space_operations f2fs_dblock_aops; extern const struct address_space_operations f2fs_node_aops; -extern const struct address_space_operations f2fs_meta_aops; extern const struct inode_operations f2fs_dir_inode_operations; extern const struct inode_operations f2fs_symlink_inode_operations; extern const struct inode_operations f2fs_encrypted_symlink_inode_operations; @@ -4626,7 +4619,7 @@ int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio); int f2fs_convert_inline_inode(struct inode *inode); int f2fs_try_convert_inline_dir(struct inode *dir, struct dentry *dentry); int f2fs_write_inline_data(struct inode *inode, struct folio *folio); -int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio); +int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entry); struct f2fs_dir_entry *f2fs_find_in_inline_dir(struct inode *dir, const struct f2fs_filename *fname, struct folio **res_folio, bool use_hash); @@ -5186,10 +5179,8 @@ static inline void f2fs_schedule_timeout_killable(long timeout, bool io) } static inline void f2fs_handle_page_eio(struct f2fs_sb_info *sbi, - struct folio *folio, enum page_type type) + pgoff_t ofs, enum page_type type) { - pgoff_t ofs = folio->index; - if (unlikely(f2fs_cp_error(sbi))) return; @@ -5224,36 +5215,10 @@ static inline bool f2fs_is_readonly(struct f2fs_sb_info *sbi) return f2fs_sb_has_readonly(sbi) || f2fs_readonly(sbi->sb); } -static inline void f2fs_truncate_meta_inode_pages(struct f2fs_sb_info *sbi, - block_t blkaddr, unsigned int cnt) -{ - bool need_submit = false; - int i = 0; - - do { - struct folio *folio; - - folio = filemap_get_folio(META_MAPPING(sbi), blkaddr + i); - if (!IS_ERR(folio)) { - if (folio_test_writeback(folio)) - need_submit = true; - f2fs_folio_put(folio, false); - } - } while (++i < cnt && !need_submit); - - if (need_submit) - f2fs_submit_merged_write_cond(sbi, sbi->meta_inode, - NULL, 0, DATA); - - truncate_inode_pages_range(META_MAPPING(sbi), - F2FS_BLK_TO_BYTES((loff_t)blkaddr), - F2FS_BLK_END_BYTES((loff_t)(blkaddr + cnt - 1))); -} - static inline void f2fs_invalidate_internal_cache(struct f2fs_sb_info *sbi, block_t blkaddr, unsigned int len) { - f2fs_truncate_meta_inode_pages(sbi, blkaddr, len); + f2fs_truncate_meta_caches(sbi, blkaddr, len); f2fs_invalidate_compress_pages_range(sbi, blkaddr, len); } diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c index a54b3ab52f1a..92daa41dd96d 100644 --- a/fs/f2fs/file.c +++ b/fs/f2fs/file.c @@ -214,7 +214,7 @@ static vm_fault_t f2fs_vm_page_mkwrite(struct vm_fault *vmf) f2fs_folio_wait_writeback(folio, DATA, false, true); - /* wait for GCed page writeback via META_MAPPING */ + /* wait for GCed page writeback via generic cache */ f2fs_wait_on_block_writeback(inode, dn.data_blkaddr); /* @@ -2563,7 +2563,7 @@ int f2fs_do_shutdown(struct f2fs_sb_info *sbi, unsigned int flag, f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_SHUTDOWN); break; case F2FS_GOING_DOWN_METAFLUSH: - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_META_IO); + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_META_IO); f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_SHUTDOWN); break; case F2FS_GOING_DOWN_NEED_FSCK: diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c index c4da2f31805b..54327cb2e27e 100644 --- a/fs/f2fs/gc.c +++ b/fs/f2fs/gc.c @@ -1066,7 +1066,7 @@ static int gc_node_segment(struct f2fs_sb_info *sbi, continue; if (phase == 0) { - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), 1, + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nid), 1, META_NAT, true); continue; } @@ -1217,7 +1217,8 @@ static int ra_data_block(struct inode *inode, pgoff_t index) struct address_space *mapping = inode->i_mapping; struct inode *atomic_inode = NULL; struct dnode_of_data dn; - struct folio *folio, *efolio; + struct folio *folio; + struct f2fs_cached_block *entry; struct f2fs_io_info fio = { .sbi = sbi, .ino = inode->i_ino, @@ -1227,6 +1228,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index) .op_flags = 0, .encrypted_page = NULL, .in_list = 0, + .is_cache = 1, }; int err = 0; @@ -1285,22 +1287,21 @@ static int ra_data_block(struct inode *inode, pgoff_t index) f2fs_wait_on_block_writeback(inode, dn.data_blkaddr); - efolio = f2fs_filemap_get_folio(META_MAPPING(sbi), dn.data_blkaddr, - FGP_LOCK | FGP_CREAT, GFP_NOFS); - if (IS_ERR(efolio)) { - err = PTR_ERR(efolio); + entry = f2fs_grab_cache(META_CACHE(sbi), dn.data_blkaddr, + F2FS_CACHE_LOCK_CREATE); + if (IS_ERR(entry)) { + err = PTR_ERR(entry); goto put_folio; } - fio.encrypted_page = &efolio->page; - - if (folio_test_uptodate(efolio)) - goto put_encrypted_page; + if (f2fs_cache_test_uptodate(entry)) + goto put_cache; - err = f2fs_submit_page_bio(&fio); + fio.cache_entry = entry; + err = f2fs_submit_cache_read(&fio); if (err) - goto put_encrypted_page; - f2fs_put_page(fio.encrypted_page, false); + goto put_cache; + f2fs_put_cache(entry, false); f2fs_folio_put(folio, true); f2fs_update_iostat(sbi, inode, FS_DATA_READ_IO, F2FS_BLKSIZE); @@ -1309,8 +1310,8 @@ static int ra_data_block(struct inode *inode, pgoff_t index) if (atomic_inode) iput(atomic_inode); return 0; -put_encrypted_page: - f2fs_put_page(fio.encrypted_page, true); +put_cache: + f2fs_put_cache(entry, true); put_folio: f2fs_folio_put(folio, true); out_iput: @@ -1320,7 +1321,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index) } /* - * Move data block via META_MAPPING while keeping locked data page. + * Move data block via meta cache while keeping locked data page. * This can be used to move blocks, aka LBAs, directly on disk. */ static int move_data_block(struct inode *inode, block_t bidx, @@ -1337,11 +1338,13 @@ static int move_data_block(struct inode *inode, block_t bidx, .op_flags = 0, .encrypted_page = NULL, .in_list = 0, + .is_cache = 1, }; struct dnode_of_data dn; struct f2fs_summary sum; struct node_info ni; - struct folio *folio, *mfolio, *efolio; + struct folio *folio; + struct f2fs_cached_block *sentry, *tentry; block_t newaddr; int err = 0; bool lfs_mode = f2fs_lfs_mode(fio.sbi); @@ -1406,20 +1409,20 @@ static int move_data_block(struct inode *inode, block_t bidx, if (lfs_mode) f2fs_down_write(&fio.sbi->io_order_lock); - mfolio = f2fs_grab_cache_folio(META_MAPPING(fio.sbi), - fio.old_blkaddr, false); - if (IS_ERR(mfolio)) { - err = PTR_ERR(mfolio); + sentry = f2fs_grab_cache(META_CACHE(fio.sbi), fio.old_blkaddr, + F2FS_CACHE_LOCK_CREATE); + if (IS_ERR(sentry)) { + err = PTR_ERR(sentry); goto up_out; } - fio.encrypted_page = folio_file_page(mfolio, fio.old_blkaddr); + fio.cache_entry = sentry; - /* read source block in mfolio */ - if (!folio_test_uptodate(mfolio)) { - err = f2fs_submit_page_bio(&fio); + /* read source block in cache */ + if (!f2fs_cache_test_uptodate(sentry)) { + err = f2fs_submit_cache_read(&fio); if (err) { - f2fs_folio_put(mfolio, true); + f2fs_put_cache(sentry, true); goto up_out; } @@ -1428,11 +1431,11 @@ static int move_data_block(struct inode *inode, block_t bidx, f2fs_update_iostat(fio.sbi, NULL, FS_GDATA_READ_IO, F2FS_BLKSIZE); - folio_lock(mfolio); - if (unlikely(!is_meta_folio(mfolio) || - !folio_test_uptodate(mfolio))) { + f2fs_lock_cache(sentry); + if (unlikely(!f2fs_is_meta_cache(sentry) || + !f2fs_cache_test_uptodate(sentry))) { err = -EIO; - f2fs_folio_put(mfolio, true); + f2fs_put_cache(sentry, true); goto up_out; } } @@ -1443,46 +1446,45 @@ static int move_data_block(struct inode *inode, block_t bidx, err = f2fs_allocate_data_block(fio.sbi, NULL, fio.old_blkaddr, &newaddr, &sum, type, NULL); if (err) { - f2fs_folio_put(mfolio, true); + f2fs_put_cache(sentry, true); /* filesystem should shutdown, no need to recovery block */ goto up_out; } - efolio = f2fs_filemap_get_folio(META_MAPPING(fio.sbi), newaddr, - FGP_LOCK | FGP_CREAT, GFP_NOFS); - if (IS_ERR(efolio)) { - err = PTR_ERR(efolio); - f2fs_folio_put(mfolio, true); + tentry = f2fs_grab_cache(META_CACHE(fio.sbi), newaddr, + F2FS_CACHE_LOCK_CREATE); + if (IS_ERR(tentry)) { + err = PTR_ERR(tentry); + f2fs_put_cache(sentry, true); goto recover_block; } - fio.encrypted_page = &efolio->page; + fio.cache_entry = tentry; /* write target block */ - f2fs_wait_on_page_writeback(fio.encrypted_page, DATA, true, true); - memcpy(page_address(fio.encrypted_page), - folio_address(mfolio), PAGE_SIZE); - f2fs_folio_put(mfolio, true); + f2fs_cache_wait_writeback_cond(tentry, DATA); + memcpy(cache_address(tentry), cache_address(sentry), PAGE_SIZE); + f2fs_put_cache(sentry, true); f2fs_invalidate_internal_cache(fio.sbi, fio.old_blkaddr, 1); - set_page_dirty(fio.encrypted_page); - if (clear_page_dirty_for_io(fio.encrypted_page)) + f2fs_mark_cache_dirty(tentry); + if (f2fs_clear_cache_dirty(tentry)) dec_page_count(fio.sbi, F2FS_DIRTY_META); - set_page_writeback(fio.encrypted_page); + f2fs_start_cache_writeback(tentry); fio.op = REQ_OP_WRITE; fio.op_flags = REQ_SYNC; fio.new_blkaddr = newaddr; - f2fs_submit_page_write(&fio); + f2fs_submit_cache_write(&fio); f2fs_update_iostat(fio.sbi, NULL, FS_GC_DATA_IO, F2FS_BLKSIZE); f2fs_update_data_blkaddr(&dn, newaddr); set_inode_flag(inode, FI_APPEND_WRITE); - f2fs_put_page(fio.encrypted_page, true); + f2fs_put_cache(tentry, true); recover_block: if (err) f2fs_do_replace_block(fio.sbi, &sum, newaddr, fio.old_blkaddr, @@ -1614,7 +1616,7 @@ static int gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum, continue; if (phase == 0) { - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), 1, + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nid), 1, META_NAT, true); continue; } @@ -1818,28 +1820,31 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, sum_blk_cnt = DIV_ROUND_UP(end_segno - segno, sbi->sums_per_block); /* readahead multi ssa blocks those have contiguous address */ if (__is_large_section(sbi)) - f2fs_ra_meta_pages(sbi, GET_SUM_BLOCK(sbi, segno), + f2fs_ra_meta_caches(sbi, GET_SUM_BLOCK(sbi, segno), sum_blk_cnt, META_SSA, true); /* reference all summary page */ while (segno < end_segno) { - struct folio *sum_folio = f2fs_get_sum_folio(sbi, segno); + struct f2fs_cached_block *sum_entry = + f2fs_get_sum_cache(sbi, segno); segno += sbi->sums_per_block; - if (IS_ERR(sum_folio)) { - int err = PTR_ERR(sum_folio); + if (IS_ERR(sum_entry)) { + int err = PTR_ERR(sum_entry); end_segno = segno - sbi->sums_per_block; segno = rounddown(start_segno, sbi->sums_per_block); while (segno < end_segno) { - sum_folio = filemap_get_folio(META_MAPPING(sbi), + sum_entry = f2fs_find_meta_cache(sbi, GET_SUM_BLOCK(sbi, segno)); - folio_put_refs(sum_folio, 2); + f2fs_put_cache(sum_entry, false); + f2fs_put_cache(sum_entry, false); segno += sbi->sums_per_block; } return err; } - folio_unlock(sum_folio); + f2fs_unlock_cache(sum_entry); + } blk_start_plug(&plug); @@ -1847,11 +1852,16 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, segno = start_segno; while (segno < end_segno) { unsigned int cur_segno; + unsigned int block_end_segno; /* find segment summary of victim */ - struct folio *sum_folio = filemap_get_folio(META_MAPPING(sbi), + struct f2fs_cached_block *sum_entry = + f2fs_find_meta_cache(sbi, GET_SUM_BLOCK(sbi, segno)); - unsigned int block_end_segno = rounddown(segno, sbi->sums_per_block) + + f2fs_bug_on(sbi, IS_ERR(sum_entry)); + + block_end_segno = rounddown(segno, sbi->sums_per_block) + sbi->sums_per_block; if (block_end_segno > end_segno) @@ -1864,8 +1874,8 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, goto next_block; } - if (!folio_test_uptodate(sum_folio) || - unlikely(f2fs_cp_error(sbi))) + if (!f2fs_cache_test_uptodate(sum_entry) || + unlikely(f2fs_cp_error(sbi))) goto next_block; for (cur_segno = segno; cur_segno < block_end_segno; @@ -1884,7 +1894,7 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, data_type = (type == SUM_TYPE_DATA) ? DATA : NODE; } - sum = SUM_BLK_PAGE_ADDR(sbi, sum_folio, cur_segno); + sum = SUM_BLK_ENTRY_ADDR(sbi, sum_entry, cur_segno); if (type != GET_SUM_TYPE(sum_footer(sbi, sum))) { f2fs_err(sbi, "Inconsistent segment (%u) type " "[%d, %d] in SIT and SSA", @@ -1926,12 +1936,14 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, cur_segno + 1 : NULL_SEGNO; if (unlikely(freezing(current))) { - folio_put_refs(sum_folio, 2); + f2fs_put_cache(sum_entry, false); + f2fs_put_cache(sum_entry, false); goto stop; } } next_block: - folio_put_refs(sum_folio, 2); + f2fs_put_cache(sum_entry, false); + f2fs_put_cache(sum_entry, false); segno = block_end_segno; } diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c index aec06fb4fd76..2156fb1fc57d 100644 --- a/fs/f2fs/inline.c +++ b/fs/f2fs/inline.c @@ -294,7 +294,7 @@ int f2fs_write_inline_data(struct inode *inode, struct folio *folio) return 0; } -int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio) +int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entry) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct f2fs_inode *ri = NULL; @@ -308,12 +308,13 @@ int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio) * x o -> remove data blocks, and then recover inline_data * x x -> recover data blocks */ - if (IS_INODE(nfolio)) - ri = F2FS_INODE(nfolio); + if (IS_INODE(cache_folio(entry))) + ri = &CACHED_NODE(entry)->i; if (f2fs_has_inline_data(inode) && ri && (ri->i_inline & F2FS_INLINE_DATA)) { struct folio *ifolio; + process_inline: ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); if (IS_ERR(ifolio)) @@ -321,7 +322,7 @@ int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio) f2fs_folio_wait_writeback(ifolio, NODE, true, true); - src_addr = inline_data_addr(inode, nfolio); + src_addr = inline_data_addr(inode, cache_folio(entry)); dst_addr = inline_data_addr(inode, ifolio); memcpy(dst_addr, src_addr, MAX_INLINE_DATA(inode)); diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c index bac1e360d966..c533da4d4d70 100644 --- a/fs/f2fs/inode.c +++ b/fs/f2fs/inode.c @@ -577,7 +577,7 @@ static int do_read_inode(struct inode *inode) static bool is_meta_ino(struct f2fs_sb_info *sbi, unsigned int ino) { - if (ino == F2FS_NODE_INO(sbi) || ino == F2FS_META_INO(sbi)) + if (ino == F2FS_NODE_INO(sbi)) return true; #ifdef CONFIG_F2FS_FS_COMPRESSION if (test_opt(sbi, COMPRESS_CACHE) && ino == F2FS_COMPRESS_INO(sbi)) @@ -624,9 +624,6 @@ struct inode *f2fs_iget(struct super_block *sb, unsigned long ino) if (ino == F2FS_NODE_INO(sbi)) { inode->i_mapping->a_ops = &f2fs_node_aops; mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS); - } else if (ino == F2FS_META_INO(sbi)) { - inode->i_mapping->a_ops = &f2fs_meta_aops; - mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS); } else if (ino == F2FS_COMPRESS_INO(sbi)) { #ifdef CONFIG_F2FS_FS_COMPRESSION inode->i_mapping->a_ops = &f2fs_compress_aops; @@ -824,8 +821,7 @@ int f2fs_write_inode(struct inode *inode, struct writeback_control *wbc) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - if (inode->i_ino == F2FS_NODE_INO(sbi) || - inode->i_ino == F2FS_META_INO(sbi)) + if (inode->i_ino == F2FS_NODE_INO(sbi)) return 0; /* @@ -919,7 +915,6 @@ static bool f2fs_pre_evict_inode(struct inode *inode) f2fs_invalidate_compress_pages(sbi, inode->i_ino); if (inode->i_ino == F2FS_NODE_INO(sbi) || - inode->i_ino == F2FS_META_INO(sbi) || inode->i_ino == F2FS_COMPRESS_INO(sbi)) return true; diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c index 46bea52e35c3..398c58fc6cad 100644 --- a/fs/f2fs/node.c +++ b/fs/f2fs/node.c @@ -140,38 +140,36 @@ static void clear_node_folio_dirty(struct folio *folio) folio_clear_uptodate(folio); } -static struct folio *get_current_nat_folio(struct f2fs_sb_info *sbi, nid_t nid) +static struct f2fs_cached_block *get_current_nat_cache(struct f2fs_sb_info *sbi, + nid_t nid) { - return f2fs_get_meta_folio_retry(sbi, current_nat_addr(sbi, nid)); + return f2fs_get_meta_cache_retry(sbi, current_nat_addr(sbi, nid)); } -static struct folio *get_next_nat_folio(struct f2fs_sb_info *sbi, nid_t nid) +static struct f2fs_cached_block *get_next_nat_cache(struct f2fs_sb_info *sbi, + nid_t nid) { - struct folio *src_folio; - struct folio *dst_folio; + struct f2fs_cached_block *src_entry; + struct f2fs_cached_block *dst_entry; pgoff_t dst_off; - void *src_addr; - void *dst_addr; struct f2fs_nm_info *nm_i = NM_I(sbi); dst_off = next_nat_addr(sbi, current_nat_addr(sbi, nid)); /* get current nat block page with lock */ - src_folio = get_current_nat_folio(sbi, nid); - if (IS_ERR(src_folio)) - return src_folio; - dst_folio = f2fs_grab_meta_folio(sbi, dst_off); - f2fs_bug_on(sbi, folio_test_dirty(src_folio)); - - src_addr = folio_address(src_folio); - dst_addr = folio_address(dst_folio); - memcpy(dst_addr, src_addr, PAGE_SIZE); - folio_mark_dirty(dst_folio); - f2fs_folio_put(src_folio, true); + src_entry = get_current_nat_cache(sbi, nid); + if (IS_ERR(src_entry)) + return src_entry; + dst_entry = f2fs_grab_meta_cache(sbi, dst_off); + f2fs_bug_on(sbi, f2fs_cache_test_dirty(src_entry)); + + memcpy(cache_address(dst_entry), cache_address(src_entry), sbi->blocksize); + f2fs_mark_cache_dirty(dst_entry); + f2fs_put_cache(src_entry, true); set_to_next_nat(nm_i, nid); - return dst_folio; + return dst_entry; } static struct nat_entry *__alloc_nat_entry(struct f2fs_sb_info *sbi, @@ -575,7 +573,7 @@ int f2fs_get_node_info(struct f2fs_sb_info *sbi, nid_t nid, struct f2fs_journal *journal = curseg->journal; nid_t start_nid = START_NID(nid); struct f2fs_nat_block *nat_blk; - struct folio *folio = NULL; + struct f2fs_cached_block *entry = NULL; struct f2fs_nat_entry ne; struct nat_entry *e; pgoff_t index; @@ -629,14 +627,14 @@ int f2fs_get_node_info(struct f2fs_sb_info *sbi, nid_t nid, index = current_nat_addr(sbi, nid); f2fs_up_read(&nm_i->nat_tree_lock); - folio = f2fs_get_meta_folio(sbi, index); - if (IS_ERR(folio)) - return PTR_ERR(folio); + entry = f2fs_get_meta_cache(sbi, index); + if (IS_ERR(entry)) + return PTR_ERR(entry); - nat_blk = folio_address(folio); + nat_blk = cache_address(entry); ne = nat_blk->entries[nid - start_nid]; node_info_from_raw_nat(ni, &ne); - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); sanity_check: if (__is_valid_data_blkaddr(ni->blk_addr) && !f2fs_is_valid_blkaddr(sbi, ni->blk_addr, @@ -1622,7 +1620,7 @@ static struct folio *__get_node_folio(struct f2fs_sb_info *sbi, pgoff_t nid, out_put_err: /* ENOENT comes from read_node_folio which is not an error. */ if (err != -ENOENT) - f2fs_handle_page_eio(sbi, folio, NODE); + f2fs_handle_page_eio(sbi, folio->index, NODE); f2fs_folio_put(folio, true); return ERR_PTR(err); } @@ -2598,6 +2596,7 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, bool sync, bool mount) { struct f2fs_nm_info *nm_i = NM_I(sbi); + struct f2fs_cached_block *entry = NULL; int i = 0, ret; nid_t nid = nm_i->next_scan_nid; @@ -2623,7 +2622,7 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, } /* readahead nat pages to be scanned */ - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), FREE_NID_PAGES, + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nid), FREE_NID_PAGES, META_NAT, true); f2fs_down_read(&nm_i->nat_tree_lock); @@ -2631,14 +2630,14 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, while (1) { if (!test_bit_le(NAT_BLOCK_OFFSET(nid), nm_i->nat_block_bitmap)) { - struct folio *folio = get_current_nat_folio(sbi, nid); + entry = get_current_nat_cache(sbi, nid); - if (IS_ERR(folio)) { - ret = PTR_ERR(folio); + if (IS_ERR(entry)) { + ret = PTR_ERR(entry); } else { - ret = scan_nat_page(sbi, folio_address(folio), + ret = scan_nat_page(sbi, cache_address(entry), nid); - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); } if (ret) { @@ -2671,7 +2670,7 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, f2fs_up_read(&nm_i->nat_tree_lock); - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nm_i->next_scan_nid), + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nm_i->next_scan_nid), nm_i->ra_nid_pages, META_NAT, false); return 0; @@ -2826,7 +2825,7 @@ int f2fs_try_to_free_nids(struct f2fs_sb_info *sbi, int nr_shrink) return nr - nr_shrink; } -int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) +int f2fs_recover_inline_xattr(struct inode *inode, struct f2fs_cached_block *entry) { void *src_addr, *dst_addr; size_t inline_size; @@ -2837,7 +2836,7 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) if (IS_ERR(ifolio)) return PTR_ERR(ifolio); - ri = F2FS_INODE(folio); + ri = &CACHED_NODE(entry)->i; if (ri->i_inline & F2FS_INLINE_XATTR) { if (!f2fs_has_inline_xattr(inode)) { set_inode_flag(inode, FI_INLINE_XATTR); @@ -2852,7 +2851,7 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) } dst_addr = inline_xattr_addr(inode, ifolio); - src_addr = inline_xattr_addr(inode, folio); + src_addr = inline_xattr_addr(inode, cache_folio(entry)); inline_size = inline_xattr_size(inode); f2fs_folio_wait_writeback(ifolio, NODE, true, true); @@ -2863,7 +2862,7 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) return 0; } -int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio) +int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); nid_t prev_xnid = F2FS_I(inode)->i_xattr_nid; @@ -2901,8 +2900,8 @@ int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio) f2fs_update_inode_page(inode); /* 3: update and set xattr node page dirty */ - if (folio) { - memcpy(F2FS_NODE(xfolio), F2FS_NODE(folio), + if (entry) { + memcpy(F2FS_NODE(xfolio), CACHED_NODE(entry), VALID_XATTR_BLOCK_SIZE); folio_mark_dirty(xfolio); } @@ -2911,10 +2910,10 @@ int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio) return 0; } -int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct folio *folio) +int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) { struct f2fs_inode *src, *dst; - nid_t ino = ino_of_node(folio); + nid_t ino = ino_of_node(cache_folio(entry)); struct node_info old_ni, new_ni; struct folio *ifolio; int err; @@ -2940,7 +2939,7 @@ int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct folio *folio) fill_node_footer(ifolio, ino, ino, 0, true); set_cold_node(ifolio, false); - src = F2FS_INODE(folio); + src = &CACHED_NODE(entry)->i; dst = F2FS_INODE(ifolio); memcpy(dst, src, offsetof(struct f2fs_inode, i_ext)); @@ -2999,24 +2998,24 @@ int f2fs_restore_node_summary(struct f2fs_sb_info *sbi, nrpages = bio_max_segs(last_offset - i); /* readahead node pages */ - f2fs_ra_meta_pages(sbi, addr, nrpages, META_POR, true); + f2fs_ra_meta_caches(sbi, addr, nrpages, META_POR, true); for (idx = addr; idx < addr + nrpages; idx++) { - struct folio *folio = f2fs_get_tmp_folio(sbi, idx); + struct f2fs_cached_block *entry = + f2fs_get_tmp_cache(sbi, idx); - if (IS_ERR(folio)) - return PTR_ERR(folio); + if (IS_ERR(entry)) + return PTR_ERR(entry); - rn = F2FS_NODE(folio); + rn = CACHED_NODE(entry); sum_entry->nid = rn->footer.nid; sum_entry->version = 0; sum_entry->ofs_in_node = 0; sum_entry++; - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); } - invalidate_mapping_pages(META_MAPPING(sbi), addr, - addr + nrpages); + f2fs_truncate_meta_caches(sbi, addr, nrpages); } return 0; } @@ -3126,7 +3125,7 @@ static int __flush_nat_entry_set(struct f2fs_sb_info *sbi, bool to_journal = true; struct f2fs_nat_block *nat_blk; struct nat_entry *ne, *cur; - struct folio *folio = NULL; + struct f2fs_cached_block *entry = NULL; /* * there are two steps to flush nat entries: @@ -3140,11 +3139,11 @@ static int __flush_nat_entry_set(struct f2fs_sb_info *sbi, if (to_journal) { down_write(&curseg->journal_rwsem); } else { - folio = get_next_nat_folio(sbi, start_nid); - if (IS_ERR(folio)) - return PTR_ERR(folio); + entry = get_next_nat_cache(sbi, start_nid); + if (IS_ERR(entry)) + return PTR_ERR(entry); - nat_blk = folio_address(folio); + nat_blk = cache_address(entry); f2fs_bug_on(sbi, !nat_blk); } @@ -3181,7 +3180,7 @@ static int __flush_nat_entry_set(struct f2fs_sb_info *sbi, up_write(&curseg->journal_rwsem); } else { __update_nat_bits(sbi, start_nid, nat_blk); - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); } /* Allow dirty nats by node block allocation in write_begin */ @@ -3252,7 +3251,7 @@ int f2fs_flush_nat_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) __has_cursum_space(sbi, journal, entry_count, NAT_JOURNAL)) continue; - f2fs_ra_meta_pages(sbi, set->set, 1, META_NAT, true); + f2fs_ra_meta_caches(sbi, set->set, 1, META_NAT, true); } /* flush dirty nats in nat entry set */ list_for_each_entry_safe(set, tmp, &sets, set_list) { @@ -3288,15 +3287,15 @@ static int __get_nat_bitmaps(struct f2fs_sb_info *sbi) nat_bits_addr = __start_cp_addr(sbi) + BLKS_PER_SEG(sbi) - nm_i->nat_bits_blocks; for (i = 0; i < nm_i->nat_bits_blocks; i++) { - struct folio *folio; + struct f2fs_cached_block *entry; - folio = f2fs_get_meta_folio(sbi, nat_bits_addr++); - if (IS_ERR(folio)) - return PTR_ERR(folio); + entry = f2fs_get_meta_cache(sbi, nat_bits_addr++); + if (IS_ERR(entry)) + return PTR_ERR(entry); memcpy(nm_i->nat_bits + F2FS_BLK_TO_BYTES(i), - folio_address(folio), F2FS_BLKSIZE); - f2fs_folio_put(folio, true); + cache_address(entry), F2FS_BLKSIZE); + f2fs_put_cache(entry, true); } cp_ver |= (cur_cp_crc(ckpt) << 32); diff --git a/fs/f2fs/node.h b/fs/f2fs/node.h index 5e114f352099..de8dcd5d4392 100644 --- a/fs/f2fs/node.h +++ b/fs/f2fs/node.h @@ -311,9 +311,9 @@ static inline void fill_node_footer_blkaddr(struct folio *folio, block_t blkaddr rn->footer.next_blkaddr = cpu_to_le32(blkaddr); } -static inline bool is_recoverable_dnode(const struct folio *folio) +static inline bool is_recoverable_dnode(struct f2fs_sb_info *sbi, const struct folio *folio) { - struct f2fs_checkpoint *ckpt = F2FS_CKPT(F2FS_F_SB(folio)); + struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); __u64 cp_ver = cur_cp_version(ckpt); /* Don't care crc part, if fsck.f2fs sets it. */ diff --git a/fs/f2fs/recovery.c b/fs/f2fs/recovery.c index aaa5227739c8..5cb56b4c1879 100644 --- a/fs/f2fs/recovery.c +++ b/fs/f2fs/recovery.c @@ -182,33 +182,33 @@ static const char *recover_printable_name(struct inode *inode, return raw->i_name; } -static int recover_dentry(struct inode *inode, struct folio *ifolio, +static int recover_dentry(struct inode *inode, struct f2fs_cached_block *entry, struct list_head *dir_list) { - struct f2fs_inode *raw_inode = F2FS_INODE(ifolio); + struct f2fs_inode *raw_inode = &CACHED_NODE(entry)->i; nid_t pino = le32_to_cpu(raw_inode->i_pino); struct f2fs_dir_entry *de; struct f2fs_filename fname; struct qstr usr_fname; struct folio *folio; struct inode *dir, *einode; - struct fsync_inode_entry *entry; + struct fsync_inode_entry *fsync_entry; int err = 0; const char *name; int name_len; - entry = get_fsync_inode(dir_list, pino); - if (!entry) { - entry = add_fsync_inode(F2FS_I_SB(inode), dir_list, + fsync_entry = get_fsync_inode(dir_list, pino); + if (!fsync_entry) { + fsync_entry = add_fsync_inode(F2FS_I_SB(inode), dir_list, pino, false); - if (IS_ERR(entry)) { - dir = ERR_CAST(entry); - err = PTR_ERR(entry); + if (IS_ERR(fsync_entry)) { + dir = ERR_CAST(fsync_entry); + err = PTR_ERR(fsync_entry); goto out; } } - dir = entry->inode; + dir = fsync_entry->inode; err = init_recovered_filename(dir, inode, raw_inode, &fname, &usr_fname); if (err) goto out; @@ -256,14 +256,14 @@ static int recover_dentry(struct inode *inode, struct folio *ifolio, out: name = recover_printable_name(inode, raw_inode, &name_len); f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, dir = %llu, err = %d", - __func__, ino_of_node(ifolio), name_len, name, + __func__, ino_of_node(cache_folio(entry)), name_len, name, IS_ERR(dir) ? 0 : dir->i_ino, err); return err; } -static int recover_quota_data(struct inode *inode, struct folio *folio) +static int recover_quota_data(struct inode *inode, struct f2fs_cached_block *entry) { - struct f2fs_inode *raw = F2FS_INODE(folio); + struct f2fs_inode *raw = &CACHED_NODE(entry)->i; struct iattr attr; uid_t i_uid = le32_to_cpu(raw->i_uid); gid_t i_gid = le32_to_cpu(raw->i_gid); @@ -300,9 +300,9 @@ static void recover_inline_flags(struct inode *inode, struct f2fs_inode *ri) clear_inode_flag(inode, FI_DATA_EXIST); } -static int recover_inode(struct inode *inode, struct folio *folio) +static int recover_inode(struct inode *inode, struct f2fs_cached_block *entry) { - struct f2fs_inode *raw = F2FS_INODE(folio); + struct f2fs_inode *raw = &CACHED_NODE(entry)->i; struct f2fs_inode_info *fi = F2FS_I(inode); const char *name; int name_len; @@ -310,7 +310,7 @@ static int recover_inode(struct inode *inode, struct folio *folio) inode->i_mode = le16_to_cpu(raw->i_mode); - err = recover_quota_data(inode, folio); + err = recover_quota_data(inode, entry); if (err) return err; @@ -357,7 +357,7 @@ static int recover_inode(struct inode *inode, struct folio *folio) name = recover_printable_name(inode, raw, &name_len); f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, inline = %x", - __func__, ino_of_node(folio), name_len, name, + __func__, ino_of_node(cache_folio(entry)), name_len, name, raw->i_inline); return 0; } @@ -386,30 +386,30 @@ static int sanity_check_node_chain(struct f2fs_sb_info *sbi, block_t blkaddr, return 0; for (i = 0; i < 2; i++) { - struct folio *folio; + struct f2fs_cached_block *entry; if (!f2fs_is_valid_blkaddr(sbi, *blkaddr_fast, META_POR)) { *is_detecting = false; return 0; } - folio = f2fs_get_tmp_folio(sbi, *blkaddr_fast); - if (IS_ERR(folio)) - return PTR_ERR(folio); + entry = f2fs_get_tmp_cache(sbi, *blkaddr_fast); + if (IS_ERR(entry)) + return PTR_ERR(entry); - if (!is_recoverable_dnode(folio)) { - f2fs_folio_put(folio, true); + if (!is_recoverable_dnode(sbi, cache_folio(entry))) { + f2fs_put_cache(entry, true); *is_detecting = false; return 0; } ra_blocks = adjust_por_ra_blocks(sbi, ra_blocks, *blkaddr_fast, - next_blkaddr_of_node(folio)); + next_blkaddr_of_node(cache_folio(entry))); - *blkaddr_fast = next_blkaddr_of_node(folio); - f2fs_folio_put(folio, true); + *blkaddr_fast = next_blkaddr_of_node(cache_folio(entry)); + f2fs_put_cache(entry, true); - f2fs_ra_meta_pages_cond(sbi, *blkaddr_fast, ra_blocks); + f2fs_ra_meta_caches_cond(sbi, *blkaddr_fast, ra_blocks); } if (*blkaddr_fast == blkaddr) { @@ -434,45 +434,47 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head, blkaddr_fast = blkaddr; while (1) { - struct fsync_inode_entry *entry; - struct folio *folio; + struct fsync_inode_entry *fsync_entry; + struct f2fs_cached_block *entry; + struct f2fs_node *rn; if (!f2fs_is_valid_blkaddr(sbi, blkaddr, META_POR)) return 0; - folio = f2fs_get_tmp_folio(sbi, blkaddr); - if (IS_ERR(folio)) { - err = PTR_ERR(folio); + entry = f2fs_get_tmp_cache(sbi, blkaddr); + if (IS_ERR(entry)) { + err = PTR_ERR(entry); break; } + rn = CACHED_NODE(entry); - if (!is_recoverable_dnode(folio)) { - f2fs_folio_put(folio, true); + if (!is_recoverable_dnode(sbi, cache_folio(entry))) { + f2fs_put_cache(entry, true); break; } - if (!is_fsync_dnode(folio)) + if (!is_fsync_dnode(cache_folio(entry))) goto next; - entry = get_fsync_inode(head, ino_of_node(folio)); - if (!entry) { + fsync_entry = get_fsync_inode(head, ino_of_node(cache_folio(entry))); + if (!fsync_entry) { bool quota_inode = false; if (!check_only && - IS_INODE(folio) && - is_dent_dnode(folio)) { - err = f2fs_recover_inode_page(sbi, folio); + IS_INODE(cache_folio(entry)) && + is_dent_dnode(cache_folio(entry))) { + err = f2fs_recover_inode_page(sbi, entry); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); break; } quota_inode = true; } - entry = add_fsync_inode(sbi, head, ino_of_node(folio), + fsync_entry = add_fsync_inode(sbi, head, ino_of_node(cache_folio(entry)), quota_inode); - if (IS_ERR(entry)) { - err = PTR_ERR(entry); + if (IS_ERR(fsync_entry)) { + err = PTR_ERR(fsync_entry); /* * CP | dnode(F) | inode(DF) * For this case, we should not give up now. @@ -482,18 +484,18 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head, *new_inode = true; goto next; } - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); break; } } - entry->blkaddr = blkaddr; + fsync_entry->blkaddr = blkaddr; - if (IS_INODE(folio) && is_dent_dnode(folio)) - entry->last_dentry = blkaddr; + if (IS_INODE(cache_folio(entry)) && is_dent_dnode(cache_folio(entry))) + fsync_entry->last_dentry = blkaddr; next: /* check next segment */ - blkaddr = next_blkaddr_of_node(folio); - f2fs_folio_put(folio, true); + blkaddr = next_blkaddr_of_node(cache_folio(entry)); + f2fs_put_cache(entry, true); err = sanity_check_node_chain(sbi, blkaddr, &blkaddr_fast, &is_detecting); @@ -519,7 +521,8 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, unsigned short blkoff = GET_BLKOFF_FROM_SEG0(sbi, blkaddr); struct f2fs_summary_block *sum_node; struct f2fs_summary sum; - struct folio *sum_folio, *node_folio; + struct f2fs_cached_block *entry = NULL; + struct folio *node_folio; struct dnode_of_data tdn = *dn; nid_t ino, nid; struct inode *inode; @@ -541,12 +544,12 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, } } - sum_folio = f2fs_get_sum_folio(sbi, segno); - if (IS_ERR(sum_folio)) - return PTR_ERR(sum_folio); - sum_node = SUM_BLK_PAGE_ADDR(sbi, sum_folio, segno); + entry = f2fs_get_sum_cache(sbi, segno); + if (IS_ERR(entry)) + return PTR_ERR(entry); + sum_node = SUM_BLK_ENTRY_ADDR(sbi, entry, segno); sum = sum_entries(sum_node)[blkoff]; - f2fs_folio_put(sum_folio, true); + f2fs_put_cache(entry, true); got_it: /* Use the locked dnode page and inode */ nid = le32_to_cpu(sum.nid); @@ -645,7 +648,7 @@ static int f2fs_reserve_new_block_retry(struct dnode_of_data *dn) } static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, - struct folio *folio) + struct f2fs_cached_block *entry) { struct dnode_of_data dn; struct node_info ni; @@ -653,19 +656,19 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, int err = 0, recovered = 0; /* step 1: recover xattr */ - if (IS_INODE(folio)) { - err = f2fs_recover_inline_xattr(inode, folio); + if (IS_INODE(cache_folio(entry))) { + err = f2fs_recover_inline_xattr(inode, entry); if (err) goto out; - } else if (f2fs_has_xattr_block(ofs_of_node(folio))) { - err = f2fs_recover_xattr_data(inode, folio); + } else if (f2fs_has_xattr_block(ofs_of_node(cache_folio(entry)))) { + err = f2fs_recover_xattr_data(inode, entry); if (!err) recovered++; goto out; } /* step 2: recover inline data */ - err = f2fs_recover_inline_data(inode, folio); + err = f2fs_recover_inline_data(inode, entry); if (err) { if (err == 1) err = 0; @@ -673,8 +676,8 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, } /* step 3: recover data indices */ - start = f2fs_start_bidx_of_node(ofs_of_node(folio), inode); - end = start + ADDRS_PER_PAGE(folio, inode); + start = f2fs_start_bidx_of_node(ofs_of_node(cache_folio(entry)), inode); + end = start + addrs_per_page(inode, IS_INODE(cache_folio(entry))); set_new_dnode(&dn, inode, NULL, NULL, 0); retry_dn: @@ -693,12 +696,12 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, if (err) goto err; - f2fs_bug_on(sbi, ni.ino != ino_of_node(folio)); + f2fs_bug_on(sbi, ni.ino != ino_of_node(cache_folio(entry))); - if (ofs_of_node(dn.node_folio) != ofs_of_node(folio)) { + if (ofs_of_node(dn.node_folio) != ofs_of_node(cache_folio(entry))) { f2fs_warn(sbi, "Inconsistent ofs_of_node, ino:%llu, ofs:%u, %u", inode->i_ino, ofs_of_node(dn.node_folio), - ofs_of_node(folio)); + ofs_of_node(cache_folio(entry))); err = -EFSCORRUPTED; f2fs_handle_error(sbi, ERROR_INCONSISTENT_FOOTER); fserror_report_file_metadata(dn.inode, err, GFP_NOFS); @@ -709,7 +712,7 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, block_t src, dest; src = f2fs_data_blkaddr(&dn); - dest = data_blkaddr(dn.inode, folio, dn.ofs_in_node); + dest = data_blkaddr(dn.inode, cache_folio(entry), dn.ofs_in_node); if (__is_valid_data_blkaddr(src) && !f2fs_is_valid_blkaddr(sbi, src, META_POR)) { @@ -784,16 +787,16 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, } } - copy_node_footer(dn.node_folio, folio); + copy_node_footer(dn.node_folio, cache_folio(entry)); fill_node_footer(dn.node_folio, dn.nid, ni.ino, - ofs_of_node(folio), false); + ofs_of_node(cache_folio(entry)), false); folio_mark_dirty(dn.node_folio); err: f2fs_put_dnode(&dn); out: f2fs_notice(sbi, "recover_data: ino = %llx, nid = %x (i_size: %s), " "range (%u, %u), recovered = %d, err = %d", - inode->i_ino, nid_of_node(folio), + inode->i_ino, nid_of_node(cache_folio(entry)), file_keep_isize(inode) ? "keep" : "recover", start, end, recovered, err); return err; @@ -820,26 +823,26 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list, blkaddr = NEXT_FREE_BLKADDR(sbi, curseg); while (1) { - struct fsync_inode_entry *entry; - struct folio *folio; + struct fsync_inode_entry *fsync_entry; + struct f2fs_cached_block *entry; if (!f2fs_is_valid_blkaddr(sbi, blkaddr, META_POR)) break; - folio = f2fs_get_tmp_folio(sbi, blkaddr); - if (IS_ERR(folio)) { - err = PTR_ERR(folio); + entry = f2fs_get_tmp_cache(sbi, blkaddr); + if (IS_ERR(entry)) { + err = PTR_ERR(entry); break; } - if (!is_recoverable_dnode(folio)) { - f2fs_folio_put(folio, true); + if (!is_recoverable_dnode(sbi, cache_folio(entry))) { + f2fs_put_cache(entry, true); break; } recoverable_dnode++; - entry = get_fsync_inode(inode_list, ino_of_node(folio)); - if (!entry) + fsync_entry = get_fsync_inode(inode_list, ino_of_node(cache_folio(entry))); + if (!fsync_entry) goto next; fsynced_dnode++; /* @@ -847,40 +850,40 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list, * In this case, we can lose the latest inode(x). * So, call recover_inode for the inode update. */ - if (IS_INODE(folio)) { - err = recover_inode(entry->inode, folio); + if (IS_INODE(cache_folio(entry))) { + err = recover_inode(fsync_entry->inode, entry); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); break; } recovered_inode++; } - if (entry->last_dentry == blkaddr) { - err = recover_dentry(entry->inode, folio, dir_list); + if (fsync_entry->last_dentry == blkaddr) { + err = recover_dentry(fsync_entry->inode, entry, dir_list); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); break; } recovered_dentry++; } - err = do_recover_data(sbi, entry->inode, folio); + err = do_recover_data(sbi, fsync_entry->inode, entry); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); break; } recovered_dnode++; - if (entry->blkaddr == blkaddr) - list_move_tail(&entry->list, tmp_inode_list); + if (fsync_entry->blkaddr == blkaddr) + list_move_tail(&fsync_entry->list, tmp_inode_list); next: ra_blocks = adjust_por_ra_blocks(sbi, ra_blocks, blkaddr, - next_blkaddr_of_node(folio)); + next_blkaddr_of_node(cache_folio(entry))); /* check next segment */ - blkaddr = next_blkaddr_of_node(folio); - f2fs_folio_put(folio, true); + blkaddr = next_blkaddr_of_node(cache_folio(entry)); + f2fs_put_cache(entry, true); - f2fs_ra_meta_pages_cond(sbi, blkaddr, ra_blocks); + f2fs_ra_meta_caches_cond(sbi, blkaddr, ra_blocks); total_dnode++; } if (!err) @@ -937,12 +940,11 @@ int f2fs_recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only) destroy_fsync_dnodes(&tmp_inode_list, err); /* truncate meta pages to be used by the recovery */ - truncate_inode_pages_range(META_MAPPING(sbi), - (loff_t)MAIN_BLKADDR(sbi) << PAGE_SHIFT, -1); - + f2fs_truncate_meta_caches(sbi, MAIN_BLKADDR(sbi), + MAX_BLKADDR(sbi) - MAIN_BLKADDR(sbi)); if (err) { truncate_inode_pages_final(NODE_MAPPING(sbi)); - truncate_inode_pages_final(META_MAPPING(sbi)); + f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); } /* diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c index 56decf9c691c..cbb3a8c9f4ab 100644 --- a/fs/f2fs/segment.c +++ b/fs/f2fs/segment.c @@ -2774,63 +2774,62 @@ int f2fs_npages_for_summary_flush(struct f2fs_sb_info *sbi, bool for_ra) return 3; } -/* - * Caller should put this summary folio - */ -struct folio *f2fs_get_sum_folio(struct f2fs_sb_info *sbi, unsigned int segno) + +struct f2fs_cached_block *f2fs_get_sum_cache(struct f2fs_sb_info *sbi, + unsigned int segno) { if (unlikely(f2fs_cp_error(sbi))) return ERR_PTR(-EIO); - return f2fs_get_meta_folio_retry(sbi, GET_SUM_BLOCK(sbi, segno)); + return f2fs_get_meta_cache_retry(sbi, GET_SUM_BLOCK(sbi, segno)); } void f2fs_update_meta_page(struct f2fs_sb_info *sbi, void *src, block_t blk_addr) { - struct folio *folio; + struct f2fs_cached_block *entry; if (!f2fs_sb_has_packed_ssa(sbi)) - folio = f2fs_grab_meta_folio(sbi, blk_addr); + entry = f2fs_grab_meta_cache(sbi, blk_addr); else - folio = f2fs_get_meta_folio_retry(sbi, blk_addr); + entry = f2fs_get_meta_cache_retry(sbi, blk_addr); - if (IS_ERR(folio)) + if (IS_ERR(entry)) return; - memcpy(folio_address(folio), src, PAGE_SIZE); - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); + memcpy(cache_address(entry), src, sbi->blocksize); + f2fs_mark_cache_dirty(entry); + f2fs_put_cache(entry, true); } static void write_sum_page(struct f2fs_sb_info *sbi, struct f2fs_summary_block *sum_blk, unsigned int segno) { - struct folio *folio; + struct f2fs_cached_block *entry; if (!f2fs_sb_has_packed_ssa(sbi)) return f2fs_update_meta_page(sbi, (void *)sum_blk, GET_SUM_BLOCK(sbi, segno)); - folio = f2fs_get_sum_folio(sbi, segno); - if (IS_ERR(folio)) + entry = f2fs_get_sum_cache(sbi, GET_SUM_BLOCK(sbi, segno)); + if (IS_ERR(entry)) return; - memcpy(SUM_BLK_PAGE_ADDR(sbi, folio, segno), sum_blk, + memcpy(SUM_BLK_ENTRY_ADDR(sbi, entry, segno), sum_blk, sbi->sum_blocksize); - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); + f2fs_mark_cache_dirty(entry); + f2fs_put_cache(entry, true); } static void write_current_sum_page(struct f2fs_sb_info *sbi, int type, block_t blk_addr) { struct curseg_info *curseg = CURSEG_I(sbi, type); - struct folio *folio = f2fs_grab_meta_folio(sbi, blk_addr); + struct f2fs_cached_block *entry = f2fs_grab_meta_cache(sbi, blk_addr); struct f2fs_summary_block *src = curseg->sum_blk; struct f2fs_summary_block *dst; - dst = folio_address(folio); - memset(dst, 0, PAGE_SIZE); + dst = cache_address(entry); + memset(dst, 0, sbi->blocksize); mutex_lock(&curseg->curseg_mutex); @@ -2843,8 +2842,8 @@ static void write_current_sum_page(struct f2fs_sb_info *sbi, mutex_unlock(&curseg->curseg_mutex); - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); + f2fs_mark_cache_dirty(entry); + f2fs_put_cache(entry, true); } static int is_next_segment_free(struct f2fs_sb_info *sbi, @@ -3160,7 +3159,7 @@ static int change_curseg(struct f2fs_sb_info *sbi, int type) struct curseg_info *curseg = CURSEG_I(sbi, type); unsigned int new_segno = curseg->next_segno; struct f2fs_summary_block *sum_node; - struct folio *sum_folio; + struct f2fs_cached_block *entry = NULL; if (curseg->inited) write_sum_page(sbi, curseg->sum_blk, curseg->segno); @@ -3176,15 +3175,15 @@ static int change_curseg(struct f2fs_sb_info *sbi, int type) curseg->alloc_type = SSR; curseg->next_blkoff = __next_free_blkoff(sbi, curseg->segno, 0); - sum_folio = f2fs_get_sum_folio(sbi, new_segno); - if (IS_ERR(sum_folio)) { + entry = f2fs_get_sum_cache(sbi, new_segno); + if (IS_ERR(entry)) { /* GC won't be able to use stale summary pages by cp_error */ memset(curseg->sum_blk, 0, sbi->sum_entry_size); - return PTR_ERR(sum_folio); + return PTR_ERR(entry); } - sum_node = SUM_BLK_PAGE_ADDR(sbi, sum_folio, new_segno); + sum_node = SUM_BLK_ENTRY_ADDR(sbi, entry, new_segno); memcpy(curseg->sum_blk, sum_node, sbi->sum_entry_size); - f2fs_folio_put(sum_folio, true); + f2fs_put_cache(entry, true); return 0; } @@ -4127,8 +4126,9 @@ static void do_write_page(struct f2fs_summary *sum, struct f2fs_io_info *fio) f2fs_up_read(&fio->sbi->io_order_lock); } -void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct folio *folio, - enum iostat_type io_type) +void f2fs_do_write_meta_cache(struct f2fs_sb_info *sbi, + struct f2fs_cached_block *entry, + enum iostat_type io_type) { struct f2fs_io_info fio = { .sbi = sbi, @@ -4136,20 +4136,21 @@ void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct folio *folio, .temp = HOT, .op = REQ_OP_WRITE, .op_flags = REQ_SYNC | REQ_META | REQ_PRIO, - .old_blkaddr = folio->index, - .new_blkaddr = folio->index, - .folio = folio, + .old_blkaddr = entry->index, + .new_blkaddr = entry->index, .encrypted_page = NULL, .in_list = 0, + .cache_entry = entry, + .is_cache = 1, }; - if (unlikely(folio->index >= MAIN_BLKADDR(sbi))) + if (unlikely(entry->index >= MAIN_BLKADDR(sbi))) fio.op_flags &= ~REQ_META; - folio_start_writeback(folio); - f2fs_submit_page_write(&fio); + f2fs_start_cache_writeback(entry); + f2fs_submit_cache_write(&fio); - stat_inc_meta_count(sbi, folio->index); + stat_inc_meta_count(sbi, entry->index); f2fs_update_iostat(sbi, NULL, io_type, F2FS_BLKSIZE); } @@ -4206,7 +4207,7 @@ int f2fs_inplace_write_data(struct f2fs_io_info *fio) } if (fio->meta_gc) - f2fs_truncate_meta_inode_pages(sbi, fio->new_blkaddr, 1); + f2fs_truncate_meta_caches(sbi, fio->new_blkaddr, 1); stat_inc_inplace_blocks(fio->sbi); @@ -4372,7 +4373,7 @@ void f2fs_folio_wait_writeback(struct folio *folio, enum page_type type, void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - struct folio *cfolio; + struct f2fs_cached_block *entry; if (!f2fs_meta_inode_gc_required(inode)) return; @@ -4380,11 +4381,12 @@ void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr) if (!__is_valid_data_blkaddr(blkaddr)) return; - cfolio = filemap_lock_folio(META_MAPPING(sbi), blkaddr); - if (!IS_ERR(cfolio)) { - f2fs_folio_wait_writeback(cfolio, DATA, true, true); - f2fs_folio_put(cfolio, true); - } + entry = f2fs_find_cache(META_CACHE(sbi), blkaddr); + if (IS_ERR(entry)) + return; + f2fs_lock_cache(entry); + f2fs_cache_wait_writeback_cond(entry, DATA); + f2fs_put_cache(entry, true); } void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr, @@ -4399,7 +4401,7 @@ void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr, for (i = 0; i < len; i++) f2fs_wait_on_block_writeback(inode, blkaddr + i); - f2fs_truncate_meta_inode_pages(sbi, blkaddr, len); + f2fs_truncate_meta_caches(sbi, blkaddr, len); } static int read_compacted_summaries(struct f2fs_sb_info *sbi) @@ -4407,16 +4409,16 @@ static int read_compacted_summaries(struct f2fs_sb_info *sbi) struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); struct curseg_info *seg_i; unsigned char *kaddr; - struct folio *folio; + struct f2fs_cached_block *entry; block_t start; int i, j, offset; start = start_sum_block(sbi); - folio = f2fs_get_meta_folio(sbi, start++); - if (IS_ERR(folio)) - return PTR_ERR(folio); - kaddr = folio_address(folio); + entry = f2fs_get_meta_cache(sbi, start++); + if (IS_ERR(entry)) + return PTR_ERR(entry); + kaddr = cache_address(entry); /* Step 1: restore nat cache */ seg_i = CURSEG_I(sbi, CURSEG_HOT_DATA); @@ -4453,16 +4455,16 @@ static int read_compacted_summaries(struct f2fs_sb_info *sbi) SUM_FOOTER_SIZE) continue; - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); - folio = f2fs_get_meta_folio(sbi, start++); - if (IS_ERR(folio)) - return PTR_ERR(folio); - kaddr = folio_address(folio); + entry = f2fs_get_meta_cache(sbi, start++); + if (IS_ERR(entry)) + return PTR_ERR(entry); + kaddr = cache_address(entry); offset = 0; } } - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); return 0; } @@ -4471,7 +4473,7 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type) struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); struct f2fs_summary_block *sum; struct curseg_info *curseg; - struct folio *new; + struct f2fs_cached_block *entry; unsigned short blk_off; unsigned int segno = 0; block_t blk_addr = 0; @@ -4498,10 +4500,10 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type) blk_addr = GET_SUM_BLOCK(sbi, segno); } - new = f2fs_get_meta_folio(sbi, blk_addr); - if (IS_ERR(new)) - return PTR_ERR(new); - sum = folio_address(new); + entry = f2fs_get_meta_cache(sbi, blk_addr); + if (IS_ERR(entry)) + return PTR_ERR(entry); + sum = cache_address(entry); if (IS_NODESEG(type)) { if (__exist_node_summaries(sbi)) { @@ -4538,7 +4540,7 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type) curseg->next_blkoff = blk_off; mutex_unlock(&curseg->curseg_mutex); out: - f2fs_folio_put(new, true); + f2fs_put_cache(entry, true); return err; } @@ -4553,8 +4555,8 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi) int npages = f2fs_npages_for_summary_flush(sbi, true); if (npages >= 2) - f2fs_ra_meta_pages(sbi, start_sum_block(sbi), npages, - META_CP, true); + f2fs_ra_meta_caches(sbi, start_sum_block(sbi), + npages, META_CP, true); /* restore for compacted data summary */ err = read_compacted_summaries(sbi); @@ -4564,7 +4566,7 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi) } if (__exist_node_summaries(sbi)) - f2fs_ra_meta_pages(sbi, + f2fs_ra_meta_caches(sbi, sum_blk_addr(sbi, NR_CURSEG_PERSIST_TYPE, type), NR_CURSEG_PERSIST_TYPE - type, META_CP, true); @@ -4587,16 +4589,16 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi) static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr) { - struct folio *folio; + struct f2fs_cached_block *entry = NULL; unsigned char *kaddr; struct f2fs_summary *summary; struct curseg_info *seg_i; int written_size = 0; int i, j; - folio = f2fs_grab_meta_folio(sbi, blkaddr++); - kaddr = folio_address(folio); - memset(kaddr, 0, PAGE_SIZE); + entry = f2fs_grab_meta_cache(sbi, blkaddr++); + kaddr = cache_address(entry); + memset(kaddr, 0, sbi->blocksize); /* Step 1: write nat cache */ seg_i = CURSEG_I(sbi, CURSEG_HOT_DATA); @@ -4612,10 +4614,10 @@ static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr) for (i = CURSEG_HOT_DATA; i <= CURSEG_COLD_DATA; i++) { seg_i = CURSEG_I(sbi, i); for (j = 0; j < f2fs_curseg_valid_blocks(sbi, i); j++) { - if (!folio) { - folio = f2fs_grab_meta_folio(sbi, blkaddr++); - kaddr = folio_address(folio); - memset(kaddr, 0, PAGE_SIZE); + if (!entry) { + entry = f2fs_grab_meta_cache(sbi, blkaddr++); + kaddr = cache_address(entry); + memset(kaddr, 0, sbi->blocksize); written_size = 0; } summary = (struct f2fs_summary *)(kaddr + written_size); @@ -4626,14 +4628,14 @@ static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr) SUM_FOOTER_SIZE) continue; - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); - folio = NULL; + f2fs_mark_cache_dirty(entry); + f2fs_put_cache(entry, true); + entry = NULL; } } - if (folio) { - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); + if (entry) { + f2fs_mark_cache_dirty(entry); + f2fs_put_cache(entry, true); } } @@ -4687,29 +4689,29 @@ int f2fs_lookup_journal_in_cursum(struct f2fs_sb_info *sbi, return -1; } -static struct folio *get_current_sit_folio(struct f2fs_sb_info *sbi, +static struct f2fs_cached_block *get_current_sit_cache(struct f2fs_sb_info *sbi, unsigned int segno) { - return f2fs_get_meta_folio(sbi, current_sit_addr(sbi, segno)); + return f2fs_get_meta_cache(sbi, current_sit_addr(sbi, segno)); } -static struct folio *get_next_sit_folio(struct f2fs_sb_info *sbi, +static struct f2fs_cached_block *get_next_sit_cache(struct f2fs_sb_info *sbi, unsigned int start) { struct sit_info *sit_i = SIT_I(sbi); - struct folio *folio; + struct f2fs_cached_block *entry; pgoff_t src_off, dst_off; src_off = current_sit_addr(sbi, start); dst_off = next_sit_addr(sbi, src_off); - folio = f2fs_grab_meta_folio(sbi, dst_off); - seg_info_to_sit_folio(sbi, folio, start); + entry = f2fs_grab_meta_cache(sbi, dst_off); + seg_info_to_sit_block(sbi, entry, start); - folio_mark_dirty(folio); + f2fs_mark_cache_dirty(entry); set_to_next_sit(sit_i, start); - return folio; + return entry; } static struct sit_entry_set *grab_sit_entry_set(void) @@ -4839,8 +4841,9 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) * #2, flush sit entries to sit page. */ list_for_each_entry_safe(ses, tmp, head, set_list) { - struct folio *folio = NULL; + struct f2fs_cached_block *entry = NULL; struct f2fs_sit_block *raw_sit = NULL; + unsigned int start_segno = ses->start_segno; unsigned int end = min(start_segno + SIT_ENTRY_PER_BLOCK, (unsigned long)MAIN_SEGS(sbi)); @@ -4854,8 +4857,8 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) if (to_journal) { down_write(&curseg->journal_rwsem); } else { - folio = get_next_sit_folio(sbi, start_segno); - raw_sit = folio_address(folio); + entry = get_next_sit_cache(sbi, start_segno); + raw_sit = cache_address(entry); } /* flush dirty sit entries in region of current sit set */ @@ -4900,7 +4903,7 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) if (to_journal) up_write(&curseg->journal_rwsem); else - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); f2fs_bug_on(sbi, ses->entry_cnt); release_sit_entry_set(ses); @@ -5091,7 +5094,7 @@ static int build_sit_entries(struct f2fs_sb_info *sbi) block_t sit_valid_blocks[2] = {0, 0}; do { - readed = f2fs_ra_meta_pages(sbi, start_blk, BIO_MAX_VECS, + readed = f2fs_ra_meta_caches(sbi, start_blk, BIO_MAX_VECS, META_SIT, true); start = start_blk * sit_i->sents_per_block; @@ -5099,15 +5102,15 @@ static int build_sit_entries(struct f2fs_sb_info *sbi) for (; start < end && start < MAIN_SEGS(sbi); start++) { struct f2fs_sit_block *sit_blk; - struct folio *folio; + struct f2fs_cached_block *entry; se = &sit_i->sentries[start]; - folio = get_current_sit_folio(sbi, start); - if (IS_ERR(folio)) - return PTR_ERR(folio); - sit_blk = folio_address(folio); + entry = get_current_sit_cache(sbi, start); + if (IS_ERR(entry)) + return PTR_ERR(entry); + sit_blk = cache_address(entry); sit = sit_blk->entries[SIT_ENTRY_OFFSET(sit_i, start)]; - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); err = check_block_count(sbi, start, &sit); if (err) diff --git a/fs/f2fs/segment.h b/fs/f2fs/segment.h index db1079169a23..fff35c63a00f 100644 --- a/fs/f2fs/segment.h +++ b/fs/f2fs/segment.h @@ -95,6 +95,8 @@ static inline void sanity_check_seg_type(struct f2fs_sb_info *sbi, #define GET_SUM_BLKOFF(sbi, segno) (segno % (sbi)->sums_per_block) #define SUM_BLK_PAGE_ADDR(sbi, folio, segno) \ (folio_address(folio) + GET_SUM_BLKOFF(sbi, segno) * (sbi)->sum_blocksize) +#define SUM_BLK_ENTRY_ADDR(sbi, entry, segno) \ + (cache_address(entry) + GET_SUM_BLKOFF(sbi, segno) * (sbi)->sum_blocksize) #define GET_SUM_TYPE(footer) ((footer)->entry_type) #define SET_SUM_TYPE(footer, type) ((footer)->entry_type = (type)) @@ -417,8 +419,8 @@ static inline void __seg_info_to_raw_sit(struct seg_entry *se, rs->mtime = cpu_to_le64(se->mtime); } -static inline void seg_info_to_sit_folio(struct f2fs_sb_info *sbi, - struct folio *folio, unsigned int start) +static inline void seg_info_to_sit_block(struct f2fs_sb_info *sbi, + struct f2fs_cached_block *entry, unsigned int start) { struct f2fs_sit_block *raw_sit; struct seg_entry *se; @@ -427,8 +429,8 @@ static inline void seg_info_to_sit_folio(struct f2fs_sb_info *sbi, (unsigned long)MAIN_SEGS(sbi)); int i; - raw_sit = folio_address(folio); - memset(raw_sit, 0, PAGE_SIZE); + raw_sit = cache_address(entry); + memset(raw_sit, 0, sbi->blocksize); for (i = 0; i < end - start; i++) { rs = &raw_sit->entries[i]; se = get_seg_entry(sbi, start + i); @@ -991,6 +993,27 @@ static inline int nr_pages_to_skip(struct f2fs_sb_info *sbi, int type) return 0; } +/* + * When writing cache asynchronously, align nr_to_write to BIO_MAX_VECS. + */ +static inline long adjust_flush_cache_number(struct f2fs_sb_info *sbi, int type) +{ + long nr_to_write; + + switch (type) { + case META: + nr_to_write = BIO_MAX_VECS; + break; + case NODE: + nr_to_write = BIO_MAX_VECS << 1; + break; + default: + f2fs_bug_on(sbi, 1); + return 0; + } + return nr_to_write; +} + /* * When writing pages, it'd better align nr_to_write for segment size. */ diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index 89affe72f4fc..d3dee9b7d999 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -1839,8 +1839,7 @@ static int f2fs_drop_inode(struct inode *inode) * drop useless meta/node dirty pages. */ if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) { - if (inode->i_ino == F2FS_NODE_INO(sbi) || - inode->i_ino == F2FS_META_INO(sbi)) { + if (inode->i_ino == F2FS_NODE_INO(sbi)) { trace_f2fs_drop_inode(inode, 1); return 1; } @@ -1942,8 +1941,7 @@ static void f2fs_dirty_inode(struct inode *inode, int flags) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - if (inode->i_ino == F2FS_NODE_INO(sbi) || - inode->i_ino == F2FS_META_INO(sbi)) + if (inode->i_ino == F2FS_NODE_INO(sbi)) return; if (is_inode_flag_set(inode, FI_AUTO_RECOVER)) @@ -2039,9 +2037,10 @@ static void f2fs_put_super(struct super_block *sb) f2fs_wait_on_all_pages(sbi, F2FS_WB_CP_DATA); - if (err || f2fs_cp_error(sbi)) { + if (err || f2fs_cp_error(sbi) || + unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) { truncate_inode_pages_final(NODE_MAPPING(sbi)); - truncate_inode_pages_final(META_MAPPING(sbi)); + f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); } f2fs_bug_on(sbi, sbi->fsync_node_num); @@ -2051,9 +2050,6 @@ static void f2fs_put_super(struct super_block *sb) iput(sbi->node_inode); sbi->node_inode = NULL; - iput(sbi->meta_inode); - sbi->meta_inode = NULL; - f2fs_destroy_cache(META_CACHE(sbi)); /* Should check the page counts after dropping all node/meta pages */ @@ -4383,7 +4379,6 @@ static void init_sb_info(struct f2fs_sb_info *sbi) sbi->allocate_section_policy = ALLOCATE_FORWARD_NOHINT; F2FS_ROOT_INO(sbi) = le32_to_cpu(raw_super->root_ino); F2FS_NODE_INO(sbi) = le32_to_cpu(raw_super->node_ino); - F2FS_META_INO(sbi) = le32_to_cpu(raw_super->meta_ino); sbi->cur_victim_sec = NULL_SECNO; sbi->gc_mode = GC_NORMAL; sbi->next_victim_seg[BG_GC] = NULL_SEGNO; @@ -5224,18 +5219,10 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) if (err) goto free_page_array_cache; - /* get an inode for meta space */ - sbi->meta_inode = f2fs_iget(sb, F2FS_META_INO(sbi)); - if (IS_ERR(sbi->meta_inode)) { - f2fs_err(sbi, "Failed to read F2FS meta data inode"); - err = PTR_ERR(sbi->meta_inode); - goto free_meta_cache; - } - err = f2fs_get_valid_checkpoint(sbi); if (err) { f2fs_err(sbi, "Failed to get valid F2FS checkpoint"); - goto free_meta_inode; + goto free_meta_cache; } if (__is_set_ckpt_flags(F2FS_CKPT(sbi), CP_QUOTA_NEED_FSCK_FLAG)) @@ -5529,7 +5516,7 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) * followed by f2fs_write_checkpoint() through f2fs_write_node_pages(), which * falls into an infinite loop in f2fs_sync_meta_pages(). */ - truncate_inode_pages_final(META_MAPPING(sbi)); + f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); /* evict some inodes being cached by GC */ evict_inodes(sb); f2fs_unregister_sysfs(sbi); @@ -5559,10 +5546,6 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) free_devices: destroy_device_list(sbi); kvfree(sbi->ckpt); -free_meta_inode: - make_bad_inode(sbi->meta_inode); - iput(sbi->meta_inode); - sbi->meta_inode = NULL; free_meta_cache: f2fs_destroy_cache(META_CACHE(sbi)); free_page_array_cache: diff --git a/include/linux/f2fs_fs.h b/include/linux/f2fs_fs.h index bb2b6cd5d507..0c027d00a1ea 100644 --- a/include/linux/f2fs_fs.h +++ b/include/linux/f2fs_fs.h @@ -35,7 +35,6 @@ #define F2FS_ROOT_INO(sbi) ((sbi)->root_ino_num) #define F2FS_NODE_INO(sbi) ((sbi)->node_ino_num) -#define F2FS_META_INO(sbi) ((sbi)->meta_ino_num) #define F2FS_COMPRESS_INO(sbi) (NM_I(sbi)->max_nid) #define F2FS_MAX_QUOTAS 3 -- 2.49.0 ^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: [f2fs-dev] [PATCH v1 05/12] f2fs: cache: use meta cache 2026-08-20 3:17 ` Chao Yu @ 2026-08-20 5:11 ` Jaegeuk Kim -1 siblings, 0 replies; 34+ messages in thread From: Jaegeuk Kim via Linux-f2fs-devel @ 2026-08-20 5:11 UTC (permalink / raw) To: Chao Yu; +Cc: linux-kernel, linux-f2fs-devel On 08/20, Chao Yu via Linux-f2fs-devel wrote: > This patch migrates F2FS meta block caching from the fake VFS inode > page cache (sbi->meta_inode) to meta cache (sbi->meta_blocks). > > It converts CP, SIT, NAT, SSA, recovery, and GC metadata I/O paths to > operate on struct f2fs_cached_block instead of folio, and removes > sbi->meta_inode. > > Signed-off-by: Chao Yu <chao@kernel.org> > --- > fs/f2fs/cache.c | 2 + > fs/f2fs/checkpoint.c | 394 ++++++++++++++++++---------------------- > fs/f2fs/compress.c | 4 +- > fs/f2fs/data.c | 29 ++- > fs/f2fs/debug.c | 12 +- > fs/f2fs/f2fs.h | 81 +++------ > fs/f2fs/file.c | 4 +- > fs/f2fs/gc.c | 134 +++++++------- > fs/f2fs/inline.c | 9 +- > fs/f2fs/inode.c | 9 +- > fs/f2fs/node.c | 125 +++++++------ > fs/f2fs/node.h | 4 +- > fs/f2fs/recovery.c | 198 ++++++++++---------- > fs/f2fs/segment.c | 201 ++++++++++---------- > fs/f2fs/segment.h | 31 +++- > fs/f2fs/super.c | 31 +--- > include/linux/f2fs_fs.h | 1 - > 17 files changed, 606 insertions(+), 663 deletions(-) > > diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c > index cb5b26046162..afef808e485a 100644 > --- a/fs/f2fs/cache.c > +++ b/fs/f2fs/cache.c > @@ -631,6 +631,8 @@ static int f2fs_cache_writeback_kthread(void *data) > break; > if (f2fs_cp_error(sbi)) > continue; > + > + f2fs_write_meta_caches(sbi); > } > return 0; > } > diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c > index 729d19680caf..1a7083540b82 100644 > --- a/fs/f2fs/checkpoint.c > +++ b/fs/f2fs/checkpoint.c > @@ -17,6 +17,7 @@ > #include <linux/delayacct.h> > #include <linux/ioprio.h> > #include <linux/math64.h> > +#include <linux/freezer.h> > > #include "f2fs.h" > #include "node.h" > @@ -235,27 +236,27 @@ struct kmem_cache *f2fs_inode_entry_slab; > /* > * We guarantee no failure on the returned page. > */ > -struct folio *f2fs_grab_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index) > +struct f2fs_cached_block *f2fs_grab_meta_cache(struct f2fs_sb_info *sbi, > + pgoff_t index) > { > - struct address_space *mapping = META_MAPPING(sbi); > - struct folio *folio; > + struct f2fs_cached_block *entry; > repeat: > - folio = f2fs_grab_cache_folio(mapping, index, false); > - if (IS_ERR(folio)) { > + entry = f2fs_grab_cache(META_CACHE(sbi), index, > + F2FS_CACHE_LOCK_CREATE); > + if (IS_ERR(entry)) { > cond_resched(); > goto repeat; > } > - f2fs_folio_wait_writeback(folio, META, true, true); > - if (!folio_test_uptodate(folio)) > - folio_mark_uptodate(folio); > - return folio; > + f2fs_cache_wait_writeback(entry); > + if (!f2fs_cache_test_uptodate(entry)) > + f2fs_cache_set_uptodate(entry); > + return entry; > } > > -static struct folio *__get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index, > - bool is_meta) > +static struct f2fs_cached_block *__get_meta_cache(struct f2fs_sb_info *sbi, > + pgoff_t index, bool is_meta) > { > - struct address_space *mapping = META_MAPPING(sbi); > - struct folio *folio; > + struct f2fs_cached_block *entry; > struct f2fs_io_info fio = { > .sbi = sbi, > .type = META, > @@ -265,70 +266,75 @@ static struct folio *__get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index, > .new_blkaddr = index, > .encrypted_page = NULL, > .is_por = !is_meta ? 1 : 0, > + .is_cache = 1, > }; > int err; > > if (unlikely(!is_meta)) > fio.op_flags &= ~REQ_META; > repeat: > - folio = f2fs_grab_cache_folio(mapping, index, false); > - if (IS_ERR(folio)) { > + entry = f2fs_grab_cache(META_CACHE(sbi), index, > + F2FS_CACHE_LOCK_CREATE); > + if (IS_ERR(entry)) { > cond_resched(); > goto repeat; > } > - if (folio_test_uptodate(folio)) > + if (f2fs_cache_test_uptodate(entry)) > goto out; > > - fio.folio = folio; > + fio.cache_entry = entry; > > - err = f2fs_submit_page_bio(&fio); > + err = f2fs_submit_cache_read(&fio); > if (err) { > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > return ERR_PTR(err); > } > > f2fs_update_iostat(sbi, NULL, FS_META_READ_IO, F2FS_BLKSIZE); > > - folio_lock(folio); > - if (unlikely(!is_meta_folio(folio))) { > - f2fs_folio_put(folio, true); > + f2fs_lock_cache(entry); > + if (unlikely(!f2fs_is_meta_cache(entry))) { > + f2fs_put_cache(entry, true); > goto repeat; > } > > - if (unlikely(!folio_test_uptodate(folio))) { > - f2fs_handle_page_eio(sbi, folio, META); > - f2fs_folio_put(folio, true); > + if (unlikely(!f2fs_cache_test_uptodate(entry))) { > + f2fs_handle_page_eio(sbi, entry->index, META); > + f2fs_put_cache(entry, true); > return ERR_PTR(-EIO); > } > out: > - return folio; > + return entry; > } > > -struct folio *f2fs_get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index) > +struct f2fs_cached_block *f2fs_get_meta_cache(struct f2fs_sb_info *sbi, > + pgoff_t index) > { > - return __get_meta_folio(sbi, index, true); > + return __get_meta_cache(sbi, index, true); > } > > -struct folio *f2fs_get_meta_folio_retry(struct f2fs_sb_info *sbi, pgoff_t index) > +struct f2fs_cached_block *f2fs_get_meta_cache_retry(struct f2fs_sb_info *sbi, > + pgoff_t index) > { > - struct folio *folio; > + struct f2fs_cached_block *entry; > int count = 0; > > retry: > - folio = __get_meta_folio(sbi, index, true); > - if (IS_ERR(folio)) { > - if (PTR_ERR(folio) == -EIO && > + entry = __get_meta_cache(sbi, index, true); > + if (IS_ERR(entry)) { > + if (PTR_ERR(entry) == -EIO && > ++count <= DEFAULT_RETRY_IO_COUNT) > goto retry; > f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_META_PAGE); > } > - return folio; > + return entry; > } > > /* for POR only */ > -struct folio *f2fs_get_tmp_folio(struct f2fs_sb_info *sbi, pgoff_t index) > +struct f2fs_cached_block *f2fs_get_tmp_cache(struct f2fs_sb_info *sbi, > + pgoff_t index) > { > - return __get_meta_folio(sbi, index, false); > + return __get_meta_cache(sbi, index, false); > } > > static bool __is_bitmap_valid(struct f2fs_sb_info *sbi, block_t blkaddr, > @@ -446,9 +452,10 @@ bool f2fs_is_valid_blkaddr_raw(struct f2fs_sb_info *sbi, > /* > * Readahead CP/NAT/SIT/SSA/POR pages > */ > -int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, > - int type, bool sync) > +int f2fs_ra_meta_caches(struct f2fs_sb_info *sbi, block_t start, > + int nrpages, int type, bool sync) > { > + struct f2fs_cached_block_list *cache = META_CACHE(sbi); > block_t blkno = start; > struct f2fs_io_info fio = { > .sbi = sbi, > @@ -458,6 +465,7 @@ int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, > .encrypted_page = NULL, > .in_list = 0, > .is_por = (type == META_POR) ? 1 : 0, > + .is_cache = 1, > }; > struct blk_plug plug; > int err; > @@ -467,7 +475,7 @@ int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, > > blk_start_plug(&plug); > for (; nrpages-- > 0; blkno++) { > - struct folio *folio; > + struct f2fs_cached_block *entry; > > if (!f2fs_is_valid_blkaddr(sbi, blkno, type)) > goto out; > @@ -494,62 +502,58 @@ int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, > fio.new_blkaddr = blkno; > break; > default: > - BUG(); > + f2fs_bug_on(sbi, 1); > } > > - folio = f2fs_grab_cache_folio(META_MAPPING(sbi), > - fio.new_blkaddr, false); > - if (IS_ERR(folio)) > + entry = f2fs_grab_cache(cache, fio.new_blkaddr, > + F2FS_CACHE_LOCK_CREATE); > + if (IS_ERR(entry)) > continue; > - if (folio_test_uptodate(folio)) { > - f2fs_folio_put(folio, true); > + if (f2fs_cache_test_uptodate(entry)) { > + f2fs_put_cache(entry, true); > continue; > } > > - fio.folio = folio; > - err = f2fs_submit_page_bio(&fio); > - f2fs_folio_put(folio, err ? true : false); > + fio.cache_entry = entry; > + err = f2fs_submit_cache_read(&fio); > + f2fs_put_cache(entry, err ? true : false); > > if (!err) > f2fs_update_iostat(sbi, NULL, FS_META_READ_IO, > - F2FS_BLKSIZE); > + sbi->blocksize); > } > out: > blk_finish_plug(&plug); > return blkno - start; > } > > -void f2fs_ra_meta_pages_cond(struct f2fs_sb_info *sbi, pgoff_t index, > - unsigned int ra_blocks) > +void f2fs_ra_meta_caches_cond(struct f2fs_sb_info *sbi, pgoff_t index, > + unsigned int ra_blocks) > { > - struct folio *folio; > + struct f2fs_cached_block *entry; > bool readahead = false; > > if (ra_blocks == RECOVERY_MIN_RA_BLOCKS) > return; > > - folio = filemap_get_folio(META_MAPPING(sbi), index); > - if (IS_ERR(folio) || !folio_test_uptodate(folio)) > + entry = f2fs_find_cache(META_CACHE(sbi), index); > + if (IS_ERR(entry) || !f2fs_cache_test_uptodate(entry)) > readahead = true; > - f2fs_folio_put(folio, false); > + f2fs_put_cache(entry, false); > > if (readahead) > - f2fs_ra_meta_pages(sbi, index, ra_blocks, META_POR, true); > + f2fs_ra_meta_caches(sbi, index, ra_blocks, META_POR, true); > } > > -static bool __f2fs_write_meta_folio(struct folio *folio, > - struct writeback_control *wbc, > +static bool __f2fs_write_meta_cache(struct f2fs_cached_block *entry, > enum iostat_type io_type) > { > - struct f2fs_sb_info *sbi = F2FS_F_SB(folio); > - > - trace_f2fs_writepage(folio, META); > + struct f2fs_sb_info *sbi = entry->cache->sbi; > > if (unlikely(f2fs_cp_error(sbi))) { > if (is_sbi_flag_set(sbi, SBI_IS_CLOSE)) { > - folio_clear_uptodate(folio); > - dec_page_count(sbi, F2FS_DIRTY_META); > - folio_unlock(folio); > + f2fs_force_clear_cache_dirty(entry); > + f2fs_unlock_cache(entry); > return true; > } > goto redirty_out; > @@ -557,10 +561,10 @@ static bool __f2fs_write_meta_folio(struct folio *folio, > if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING))) > goto redirty_out; > > - f2fs_do_write_meta_page(sbi, folio, io_type); > + f2fs_do_write_meta_cache(sbi, entry, io_type); > dec_page_count(sbi, F2FS_DIRTY_META); > > - folio_unlock(folio); > + f2fs_unlock_cache(entry); > > if (unlikely(f2fs_cp_error(sbi))) > f2fs_submit_merged_write(sbi, META); > @@ -568,101 +572,89 @@ static bool __f2fs_write_meta_folio(struct folio *folio, > return true; > > redirty_out: > - folio_redirty_for_writepage(wbc, folio); > + f2fs_cache_set_dirty(entry); > return false; > } > > -static int f2fs_write_meta_pages(struct address_space *mapping, > - struct writeback_control *wbc) > +void f2fs_write_meta_caches(struct f2fs_sb_info *sbi) > { > - struct f2fs_sb_info *sbi = F2FS_M_SB(mapping); > struct f2fs_lock_context lc; > - long diff, written; > + long nr_to_write = LONG_MAX; > > if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING))) > - goto skip_write; > + return; > > - /* collect a number of dirty meta pages and write together */ > - if (wbc->sync_mode != WB_SYNC_ALL && > - get_pages(sbi, F2FS_DIRTY_META) < > - nr_pages_to_skip(sbi, META)) > - goto skip_write; > + /* collect a number of dirty meta caches and write together */ > + if (get_pages(sbi, F2FS_DIRTY_META) < > + nr_pages_to_skip(sbi, META)) > + return; > > - /* if locked failed, cp will flush dirty pages instead */ > + /* if locked failed, cp will flush dirty caches instead */ > if (!f2fs_down_write_trylock_trace(&sbi->cp_global_sem, &lc)) > - goto skip_write; > + return; > > - trace_f2fs_writepages(mapping->host, wbc, META); > - diff = nr_pages_to_write(sbi, META, wbc); > - written = f2fs_sync_meta_pages(sbi, wbc->nr_to_write, FS_META_IO); > + nr_to_write = adjust_flush_cache_number(sbi, META); > + f2fs_sync_meta_caches(sbi, nr_to_write, false, FS_META_IO); > f2fs_up_write_trace(&sbi->cp_global_sem, &lc); > - wbc->nr_to_write = max((long)0, wbc->nr_to_write - written - diff); > - return 0; > - > -skip_write: > - wbc->pages_skipped += get_pages(sbi, F2FS_DIRTY_META); > - trace_f2fs_writepages(mapping->host, wbc, META); > - return 0; > } > > -long f2fs_sync_meta_pages(struct f2fs_sb_info *sbi, long nr_to_write, > - enum iostat_type io_type) > +long f2fs_sync_meta_caches(struct f2fs_sb_info *sbi, long nr_to_write, > + bool sync, enum iostat_type io_type) > { > - struct address_space *mapping = META_MAPPING(sbi); > pgoff_t index = 0, prev = ULONG_MAX; > - struct folio_batch fbatch; > + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; > long nwritten = 0; > - int nr_folios; > - struct writeback_control wbc = {}; > + int nr; > struct blk_plug plug; > - > - folio_batch_init(&fbatch); > + bool background = nr_to_write != LONG_MAX; > > blk_start_plug(&plug); > > - while ((nr_folios = filemap_get_folios_tag(mapping, &index, > - (pgoff_t)-1, > - PAGECACHE_TAG_DIRTY, &fbatch))) { > + while ((nr = f2fs_cache_gang_lookup_tag(META_CACHE(sbi), entries, > + &index, F2FS_ONSTACK_CACHES, F2FS_CACHE_TAG_DIRTY))) { > int i; > > - for (i = 0; i < nr_folios; i++) { > - struct folio *folio = fbatch.folios[i]; > + for (i = 0; i < nr; i++) { > + struct f2fs_cached_block *entry = entries[i]; > + > + if (background && unlikely(freezing(current))) { > + f2fs_cache_gang_release(entries, nr); > + goto stop; > + } > > - if (nr_to_write != LONG_MAX && i != 0 && > - folio->index != prev + > - folio_nr_pages(fbatch.folios[i-1])) { > - folio_batch_release(&fbatch); > + if (background && i != 0 && > + entry->index != prev + 1) { > + f2fs_cache_gang_release(entries, nr); > goto stop; > } > > - folio_lock(folio); > + f2fs_lock_cache(entry); > > - if (unlikely(!is_meta_folio(folio))) { > + if (unlikely(!f2fs_is_meta_cache(entry))) { > continue_unlock: > - folio_unlock(folio); > + f2fs_unlock_cache(entry); > continue; > } > - if (!folio_test_dirty(folio)) { > + if (!f2fs_cache_test_dirty(entry)) { > /* someone wrote it for us */ > goto continue_unlock; > } > > - f2fs_folio_wait_writeback(folio, META, true, true); > + f2fs_cache_wait_writeback(entry); > > - if (!folio_clear_dirty_for_io(folio)) > + if (!f2fs_clear_cache_dirty(entry)) > goto continue_unlock; > > - if (!__f2fs_write_meta_folio(folio, &wbc, > - io_type)) { > - folio_unlock(folio); > + if (!__f2fs_write_meta_cache(entry, io_type)) { > + f2fs_unlock_cache(entry); > break; > } > - nwritten += folio_nr_pages(folio); > - prev = folio->index; > + nwritten += sbi->blocksize; > + prev = entry->index; > if (unlikely(nwritten >= nr_to_write)) > break; > } > - folio_batch_release(&fbatch); > + f2fs_cache_gang_release(entries, nr); > cond_resched(); > } > stop: > @@ -674,29 +666,6 @@ long f2fs_sync_meta_pages(struct f2fs_sb_info *sbi, long nr_to_write, > return nwritten; > } > > -static bool f2fs_dirty_meta_folio(struct address_space *mapping, > - struct folio *folio) > -{ > - trace_f2fs_set_page_dirty(folio, META); > - > - if (!folio_test_uptodate(folio)) > - folio_mark_uptodate(folio); > - if (filemap_dirty_folio(mapping, folio)) { > - inc_page_count(F2FS_M_SB(mapping), F2FS_DIRTY_META); > - folio_set_f2fs_reference(folio); > - return true; > - } > - return false; > -} > - > -const struct address_space_operations f2fs_meta_aops = { > - .writepages = f2fs_write_meta_pages, > - .dirty_folio = f2fs_dirty_meta_folio, > - .invalidate_folio = f2fs_invalidate_folio, > - .release_folio = f2fs_release_folio, > - .migrate_folio = filemap_migrate_folio, > -}; > - > static void __add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, > unsigned int devidx, int type) > { > @@ -1036,20 +1005,20 @@ int f2fs_recover_orphan_inodes(struct f2fs_sb_info *sbi) > start_blk = __start_cp_addr(sbi) + 1 + __cp_payload(sbi); > orphan_blocks = __start_sum_addr(sbi) - 1 - __cp_payload(sbi); > > - f2fs_ra_meta_pages(sbi, start_blk, orphan_blocks, META_CP, true); > + f2fs_ra_meta_caches(sbi, start_blk, orphan_blocks, META_CP, true); > > for (i = 0; i < orphan_blocks; i++) { > - struct folio *folio; > + struct f2fs_cached_block *entry; > struct f2fs_orphan_block *orphan_blk; > unsigned int entry_count; > > - folio = f2fs_get_meta_folio(sbi, start_blk + i); > - if (IS_ERR(folio)) { > - err = PTR_ERR(folio); > + entry = f2fs_get_meta_cache(sbi, start_blk + i); > + if (IS_ERR(entry)) { > + err = PTR_ERR(entry); > goto out; > } > > - orphan_blk = folio_address(folio); > + orphan_blk = cache_address(entry); > entry_count = le32_to_cpu(orphan_blk->entry_count); > if (entry_count > F2FS_ORPHANS_PER_BLOCK) { > f2fs_err(sbi, "invalid orphan inode entry count %u", > @@ -1057,7 +1026,7 @@ int f2fs_recover_orphan_inodes(struct f2fs_sb_info *sbi) > set_sbi_flag(sbi, SBI_NEED_FSCK); > f2fs_handle_error(sbi, ERROR_INCONSISTENT_ORPHAN); > err = -EFSCORRUPTED; > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > goto out; > } > > @@ -1066,11 +1035,11 @@ int f2fs_recover_orphan_inodes(struct f2fs_sb_info *sbi) > > err = recover_orphan_inode(sbi, ino); > if (err) { > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > goto out; > } > } > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > } > /* clear Orphan Flag */ > clear_ckpt_flags(sbi, CP_ORPHAN_PRESENT_FLAG); > @@ -1087,9 +1056,9 @@ static void write_orphan_inodes(struct f2fs_sb_info *sbi, block_t start_blk) > unsigned int nentries = 0; > unsigned short index = 1; > unsigned short orphan_blocks; > - struct folio *folio = NULL; > struct ino_entry *orphan = NULL; > struct inode_management *im = &sbi->im[ORPHAN_INO]; > + struct f2fs_cached_block *entry = NULL; > > orphan_blocks = GET_ORPHAN_BLOCKS(im->ino_num); > > @@ -1102,9 +1071,9 @@ static void write_orphan_inodes(struct f2fs_sb_info *sbi, block_t start_blk) > > /* loop for each orphan inode entry and write them in journal block */ > list_for_each_entry(orphan, head, list) { > - if (!folio) { > - folio = f2fs_grab_meta_folio(sbi, start_blk++); > - orphan_blk = folio_address(folio); > + if (!entry) { > + entry = f2fs_grab_meta_cache(sbi, start_blk++); > + orphan_blk = (struct f2fs_orphan_block *)cache_address(entry); > memset(orphan_blk, 0, sizeof(*orphan_blk)); > } > > @@ -1119,20 +1088,20 @@ static void write_orphan_inodes(struct f2fs_sb_info *sbi, block_t start_blk) > orphan_blk->blk_addr = cpu_to_le16(index); > orphan_blk->blk_count = cpu_to_le16(orphan_blocks); > orphan_blk->entry_count = cpu_to_le32(nentries); > - folio_mark_dirty(folio); > - f2fs_folio_put(folio, true); > + f2fs_mark_cache_dirty(entry); > + f2fs_put_cache(entry, true); > index++; > nentries = 0; > - folio = NULL; > + entry = NULL; > } > } > > - if (folio) { > + if (entry) { > orphan_blk->blk_addr = cpu_to_le16(index); > orphan_blk->blk_count = cpu_to_le16(orphan_blocks); > orphan_blk->entry_count = cpu_to_le32(nentries); > - folio_mark_dirty(folio); > - f2fs_folio_put(folio, true); > + f2fs_mark_cache_dirty(entry); > + f2fs_put_cache(entry, true); > } > } > > @@ -1151,29 +1120,29 @@ static __u32 f2fs_checkpoint_chksum(struct f2fs_checkpoint *ckpt) > } > > static int get_checkpoint_version(struct f2fs_sb_info *sbi, block_t cp_addr, > - struct f2fs_checkpoint **cp_block, struct folio **cp_folio, > + struct f2fs_checkpoint **cp_block, struct f2fs_cached_block **cp_entry, > unsigned long long *version) > { > size_t crc_offset = 0; > __u32 crc; > > - *cp_folio = f2fs_get_meta_folio(sbi, cp_addr); > - if (IS_ERR(*cp_folio)) > - return PTR_ERR(*cp_folio); > + *cp_entry = f2fs_get_meta_cache(sbi, cp_addr); > + if (IS_ERR(*cp_entry)) > + return PTR_ERR(*cp_entry); > > - *cp_block = folio_address(*cp_folio); > + *cp_block = (struct f2fs_checkpoint *)cache_address(*cp_entry); > > crc_offset = le32_to_cpu((*cp_block)->checksum_offset); > if (crc_offset < CP_MIN_CHKSUM_OFFSET || > crc_offset > CP_CHKSUM_OFFSET) { > - f2fs_folio_put(*cp_folio, true); > + f2fs_put_cache(*cp_entry, true); > f2fs_warn(sbi, "invalid crc_offset: %zu", crc_offset); > return -EINVAL; > } > > crc = f2fs_checkpoint_chksum(*cp_block); > if (crc != cur_cp_crc(*cp_block)) { > - f2fs_folio_put(*cp_folio, true); > + f2fs_put_cache(*cp_entry, true); > f2fs_warn(sbi, "invalid crc value"); > return -EINVAL; > } > @@ -1182,17 +1151,17 @@ static int get_checkpoint_version(struct f2fs_sb_info *sbi, block_t cp_addr, > return 0; > } > > -static struct folio *validate_checkpoint(struct f2fs_sb_info *sbi, > +static struct f2fs_cached_block *validate_checkpoint(struct f2fs_sb_info *sbi, > block_t cp_addr, unsigned long long *version) > { > - struct folio *cp_folio_1 = NULL, *cp_folio_2 = NULL; > + struct f2fs_cached_block *cp_entry_1 = NULL, *cp_entry_2 = NULL; > struct f2fs_checkpoint *cp_block = NULL; > unsigned long long cur_version = 0, pre_version = 0; > unsigned int cp_blocks; > int err; > > err = get_checkpoint_version(sbi, cp_addr, &cp_block, > - &cp_folio_1, version); > + &cp_entry_1, version); > if (err) > return NULL; > > @@ -1207,19 +1176,20 @@ static struct folio *validate_checkpoint(struct f2fs_sb_info *sbi, > > cp_addr += cp_blocks - 1; > err = get_checkpoint_version(sbi, cp_addr, &cp_block, > - &cp_folio_2, version); > + &cp_entry_2, version); > if (err) > goto invalid_cp; > cur_version = *version; > > if (cur_version == pre_version) { > *version = cur_version; > - f2fs_folio_put(cp_folio_2, true); > - return cp_folio_1; > + f2fs_put_cache(cp_entry_2, true); > + return cp_entry_1; > } > - f2fs_folio_put(cp_folio_2, true); > + f2fs_put_cache(cp_entry_2, true); > invalid_cp: > - f2fs_folio_put(cp_folio_1, true); > + if (cp_entry_1) > + f2fs_put_cache(cp_entry_1, true); > return NULL; > } > > @@ -1227,7 +1197,7 @@ int f2fs_get_valid_checkpoint(struct f2fs_sb_info *sbi) > { > struct f2fs_checkpoint *cp_block; > struct f2fs_super_block *fsb = sbi->raw_super; > - struct folio *cp1, *cp2, *cur_folio; > + struct f2fs_cached_block *cp1 = NULL, *cp2 = NULL, *cur_entry; > unsigned long blk_size = sbi->blocksize; > unsigned long long cp1_version = 0, cp2_version = 0; > unsigned long long cp_start_blk_no; > @@ -1254,22 +1224,22 @@ int f2fs_get_valid_checkpoint(struct f2fs_sb_info *sbi) > > if (cp1 && cp2) { > if (ver_after(cp2_version, cp1_version)) > - cur_folio = cp2; > + cur_entry = cp2; > else > - cur_folio = cp1; > + cur_entry = cp1; > } else if (cp1) { > - cur_folio = cp1; > + cur_entry = cp1; > } else if (cp2) { > - cur_folio = cp2; > + cur_entry = cp2; > } else { > err = -EFSCORRUPTED; > goto fail_no_cp; > } > > - cp_block = folio_address(cur_folio); > + cp_block = (struct f2fs_checkpoint *)cache_address(cur_entry); > memcpy(sbi->ckpt, cp_block, blk_size); > > - if (cur_folio == cp1) > + if (cur_entry == cp1) > sbi->cur_cp_pack = 1; > else > sbi->cur_cp_pack = 2; > @@ -1284,30 +1254,35 @@ int f2fs_get_valid_checkpoint(struct f2fs_sb_info *sbi) > goto done; > > cp_blk_no = le32_to_cpu(fsb->cp_blkaddr); > - if (cur_folio == cp2) > + if (cur_entry == cp2) > cp_blk_no += BIT(le32_to_cpu(fsb->log_blocks_per_seg)); > > for (i = 1; i < cp_blks; i++) { > + struct f2fs_cached_block *cur_entry_payload; > void *sit_bitmap_ptr; > unsigned char *ckpt = (unsigned char *)sbi->ckpt; > > - cur_folio = f2fs_get_meta_folio(sbi, cp_blk_no + i); > - if (IS_ERR(cur_folio)) { > - err = PTR_ERR(cur_folio); > + cur_entry_payload = f2fs_get_meta_cache(sbi, cp_blk_no + i); > + if (IS_ERR(cur_entry_payload)) { > + err = PTR_ERR(cur_entry_payload); > goto free_fail_no_cp; > } > - sit_bitmap_ptr = folio_address(cur_folio); > + sit_bitmap_ptr = cache_address(cur_entry_payload); > memcpy(ckpt + i * blk_size, sit_bitmap_ptr, blk_size); > - f2fs_folio_put(cur_folio, true); > + f2fs_put_cache(cur_entry_payload, true); > } > done: > - f2fs_folio_put(cp1, true); > - f2fs_folio_put(cp2, true); > + if (cp1) > + f2fs_put_cache(cp1, true); In f2fs_put_cache(x, y), if (!x) return; > + if (cp2) > + f2fs_put_cache(cp2, true); > return 0; > > free_fail_no_cp: > - f2fs_folio_put(cp1, true); > - f2fs_folio_put(cp2, true); > + if (cp1) > + f2fs_put_cache(cp1, true); > + if (cp2) > + f2fs_put_cache(cp2, true); > fail_no_cp: > kvfree(sbi->ckpt); > return err; > @@ -1621,7 +1596,7 @@ void f2fs_wait_on_all_pages(struct f2fs_sb_info *sbi, int type) > break; > > if (type == F2FS_DIRTY_META) > - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_CP_META_IO); > + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_META_IO); > else if (type == F2FS_WB_CP_DATA) > f2fs_submit_merged_write(sbi, DATA); > > @@ -1700,31 +1675,24 @@ static void update_ckpt_flags(struct f2fs_sb_info *sbi, struct cp_control *cpc) > static void commit_checkpoint(struct f2fs_sb_info *sbi, > void *src, block_t blk_addr) > { > - struct writeback_control wbc = {}; > - > - /* > - * filemap_get_folios_tag and folio_lock again will take > - * some extra time. Therefore, f2fs_update_meta_pages and > - * f2fs_sync_meta_pages are combined in this function. > - */ > - struct folio *folio = f2fs_grab_meta_folio(sbi, blk_addr); > + struct f2fs_cached_block *entry = f2fs_grab_meta_cache(sbi, blk_addr); > > - memcpy(folio_address(folio), src, PAGE_SIZE); > + memcpy(cache_address(entry), src, F2FS_BLKSIZE); > > - folio_mark_dirty(folio); > - if (unlikely(!folio_clear_dirty_for_io(folio))) > + f2fs_mark_cache_dirty(entry); > + if (unlikely(!f2fs_clear_cache_dirty(entry))) > f2fs_bug_on(sbi, 1); > > /* writeout cp pack 2 page */ > - if (unlikely(!__f2fs_write_meta_folio(folio, &wbc, FS_CP_META_IO))) { > + if (unlikely(!__f2fs_write_meta_cache(entry, FS_CP_META_IO))) { > if (f2fs_cp_error(sbi)) { > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > return; > } > f2fs_bug_on(sbi, true); > } > > - f2fs_folio_put(folio, false); > + f2fs_put_cache(entry, false); > > /* submit checkpoint (with barrier if NOBARRIER is not set) */ > f2fs_submit_merged_write(sbi, META_FLUSH); > @@ -1793,7 +1761,7 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc) > int err; > > /* Flush all the NAT/SIT pages */ > - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_CP_META_IO); > + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_CP_META_IO); > > stat_cp_time(cpc, CP_TIME_SYNC_META); > > @@ -1892,7 +1860,7 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc) > } > > /* Here, we have one bio having CP pack except cp pack 2 page */ > - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_CP_META_IO); > + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_CP_META_IO); > stat_cp_time(cpc, CP_TIME_SYNC_CP_META); > > /* Wait for all dirty meta pages to be submitted for IO */ > @@ -1919,10 +1887,10 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc) > * used for migration of encrypted, verity or compressed inode's blocks. > */ > if (f2fs_sb_has_encrypt(sbi) || f2fs_sb_has_verity(sbi) || > - f2fs_sb_has_compression(sbi)) > - f2fs_bug_on(sbi, > - invalidate_inode_pages2_range(META_MAPPING(sbi), > - MAIN_BLKADDR(sbi), MAX_BLKADDR(sbi) - 1)); > + f2fs_sb_has_compression(sbi)) { > + f2fs_truncate_meta_caches(sbi, MAIN_BLKADDR(sbi), > + MAX_BLKADDR(sbi) - MAIN_BLKADDR(sbi)); > + } > > f2fs_release_ino_entry(sbi, false); > > diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c > index 91855d91bbdd..bb749f6257a1 100644 > --- a/fs/f2fs/compress.c > +++ b/fs/f2fs/compress.c > @@ -1150,7 +1150,7 @@ static int prepare_compress_overwrite(struct compress_ctx *cc, > f2fs_compress_ctx_add_page(cc, folio); > > if (!folio_test_uptodate(folio)) { > - f2fs_handle_page_eio(sbi, folio, DATA); > + f2fs_handle_page_eio(sbi, folio->index, DATA); > release_and_retry: > f2fs_put_rpages(cc); > f2fs_unlock_rpages(cc, i + 1); > @@ -1359,7 +1359,7 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc, > fio.old_blkaddr = data_blkaddr(dn.inode, dn.node_folio, > dn.ofs_in_node + i + 1); > > - /* wait for GCed page writeback via META_MAPPING */ > + /* wait for GCed page writeback via generic cache */ > f2fs_wait_on_block_writeback(inode, fio.old_blkaddr); > > if (fio.encrypted) { > diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c > index 110282bb8dcd..92c3293f0a1e 100644 > --- a/fs/f2fs/data.c > +++ b/fs/f2fs/data.c > @@ -65,8 +65,7 @@ bool f2fs_is_cp_guaranteed(const struct folio *folio) > > inode = mapping->host; > > - if (inode->i_ino == F2FS_META_INO(sbi) || > - inode->i_ino == F2FS_NODE_INO(sbi) || > + if (inode->i_ino == F2FS_NODE_INO(sbi) || > S_ISDIR(inode->i_mode)) > return true; > > @@ -84,9 +83,6 @@ static enum count_type __read_io_type(struct folio *folio) > struct inode *inode = mapping->host; > struct f2fs_sb_info *sbi = F2FS_I_SB(inode); > > - if (inode->i_ino == F2FS_META_INO(sbi)) > - return F2FS_RD_META; > - > if (inode->i_ino == F2FS_NODE_INO(sbi)) > return F2FS_RD_NODE; > } > @@ -1452,7 +1448,7 @@ static void f2fs_submit_page_read(struct inode *inode, struct fsverity_info *vi, > bio = f2fs_grab_read_bio(inode, vi, blkaddr, 1, op_flags, folio->index, > for_write); > > - /* wait for GCed page writeback via META_MAPPING */ > + /* wait for GCed page writeback via generic cache */ > f2fs_wait_on_block_writeback(inode, blkaddr); > > if (!bio_add_folio(bio, folio, PAGE_SIZE, 0)) > @@ -3110,7 +3106,7 @@ static void f2fs_readahead(struct readahead_control *rac) > int f2fs_encrypt_one_page(struct f2fs_io_info *fio) > { > struct inode *inode = fio_inode(fio); > - struct folio *mfolio; > + struct f2fs_cached_block *entry; > struct page *page; > > if (!f2fs_encrypted_file(inode)) > @@ -3126,12 +3122,13 @@ int f2fs_encrypt_one_page(struct f2fs_io_info *fio) > if (IS_ERR(fio->encrypted_page)) > return PTR_ERR(fio->encrypted_page); > > - mfolio = filemap_lock_folio(META_MAPPING(fio->sbi), fio->old_blkaddr); > - if (!IS_ERR(mfolio)) { > - if (folio_test_uptodate(mfolio)) > - memcpy(folio_address(mfolio), > - page_address(fio->encrypted_page), PAGE_SIZE); > - f2fs_folio_put(mfolio, true); > + entry = f2fs_find_cache(META_CACHE(fio->sbi), fio->old_blkaddr); > + if (!IS_ERR(entry)) { > + f2fs_lock_cache(entry); > + if (f2fs_cache_test_uptodate(entry)) > + memcpy(cache_address(entry), page_address(fio->encrypted_page), > + F2FS_BLKSIZE); > + f2fs_put_cache(entry, true); > } > return 0; > } > @@ -3297,7 +3294,7 @@ int f2fs_do_write_data_page(struct f2fs_io_info *fio) > goto out_writepage; > } > > - /* wait for GCed page writeback via META_MAPPING */ > + /* wait for GCed page writeback via generic cache */ > if (fio->meta_gc) > f2fs_wait_on_block_writeback(inode, fio->old_blkaddr); > > @@ -4380,9 +4377,7 @@ void f2fs_invalidate_folio(struct folio *folio, size_t offset, size_t length) > return; > > if (folio_test_dirty(folio)) { > - if (inode->i_ino == F2FS_META_INO(sbi)) { > - dec_page_count(sbi, F2FS_DIRTY_META); > - } else if (inode->i_ino == F2FS_NODE_INO(sbi)) { > + if (inode->i_ino == F2FS_NODE_INO(sbi)) { > dec_page_count(sbi, F2FS_DIRTY_NODES); > } else { > inode_dec_dirty_pages(inode); > diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c > index ff379aff4472..14059a50739c 100644 > --- a/fs/f2fs/debug.c > +++ b/fs/f2fs/debug.c > @@ -224,8 +224,7 @@ static void update_general_status(struct f2fs_sb_info *sbi) > si->dirty_count = dirty_segments(sbi); > if (sbi->node_inode) > si->node_pages = NODE_MAPPING(sbi)->nrpages; > - if (sbi->meta_inode) > - si->meta_pages = META_MAPPING(sbi)->nrpages; > + si->meta_caches = META_CACHE(sbi)->num_entries; > #ifdef CONFIG_F2FS_FS_COMPRESSION > if (sbi->compress_inode) { > si->compress_pages = COMPRESS_MAPPING(sbi)->nrpages; > @@ -388,11 +387,8 @@ static void update_mem_info(struct f2fs_sb_info *sbi) > > si->page_mem += (unsigned long long)npages << PAGE_SHIFT; > } > - if (sbi->meta_inode) { > - unsigned long npages = META_MAPPING(sbi)->nrpages; > - > - si->page_mem += (unsigned long long)npages << PAGE_SHIFT; > - } > + si->page_mem += (unsigned long long)META_CACHE(sbi)->num_entries << PAGE_SHIFT; > + si->cache_mem += META_CACHE(sbi)->num_entries * sizeof(struct f2fs_cached_block); > #ifdef CONFIG_F2FS_FS_COMPRESSION > if (sbi->compress_inode) { > unsigned long npages = COMPRESS_MAPPING(sbi)->nrpages; > @@ -708,7 +704,7 @@ static int stat_show(struct seq_file *s, void *v) > seq_printf(s, " - quota data: %4d in quota files:%4d\n", > si->ndirty_qdata, si->nquota_files); > seq_printf(s, " - meta: %4d in %4d\n", > - si->ndirty_meta, si->meta_pages); > + si->ndirty_meta, si->meta_caches); > seq_printf(s, " - imeta: %4d\n", > si->ndirty_imeta); > seq_printf(s, " - fsync mark: %4lld\n", > diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h > index 6e20b3586f26..9a353dcd6658 100644 > --- a/fs/f2fs/f2fs.h > +++ b/fs/f2fs/f2fs.h > @@ -1828,7 +1828,6 @@ struct f2fs_sb_info { > struct f2fs_checkpoint *ckpt; /* raw checkpoint pointer */ > int cur_cp_pack; /* remain current cp pack */ > spinlock_t cp_lock; /* for flag in ckpt */ > - struct inode *meta_inode; /* cache meta blocks */ > struct f2fs_rwsem cp_global_sem; /* checkpoint procedure lock */ > struct f2fs_rwsem cp_rwsem; /* blocking FS operations */ > struct f2fs_rwsem node_write; /* locking node writes */ > @@ -1873,7 +1872,6 @@ struct f2fs_sb_info { > unsigned int blocksize; /* block size */ > unsigned int root_ino_num; /* root inode number*/ > unsigned int node_ino_num; /* node inode number*/ > - unsigned int meta_ino_num; /* meta inode number*/ > unsigned int log_blocks_per_seg; /* log2 blocks per segment */ > unsigned int blocks_per_seg; /* blocks per segment */ > unsigned int segs_per_sec; /* segments per section */ > @@ -2319,9 +2317,9 @@ static inline struct dirty_seglist_info *DIRTY_I(struct f2fs_sb_info *sbi) > return (struct dirty_seglist_info *)(SM_I(sbi)->dirty_info); > } > > -static inline struct address_space *META_MAPPING(struct f2fs_sb_info *sbi) > +static inline bool f2fs_is_meta_cache(struct f2fs_cached_block *entry) > { > - return sbi->meta_inode->i_mapping; > + return entry->cache && entry->cache == META_CACHE(entry->cache->sbi); > } > > static inline struct address_space *NODE_MAPPING(struct f2fs_sb_info *sbi) > @@ -2329,11 +2327,6 @@ static inline struct address_space *NODE_MAPPING(struct f2fs_sb_info *sbi) > return sbi->node_inode->i_mapping; > } > > -static inline bool is_meta_folio(struct folio *folio) > -{ > - return folio->mapping == META_MAPPING(F2FS_F_SB(folio)); > -} > - > static inline bool is_node_folio(struct folio *folio) > { > return folio->mapping == NODE_MAPPING(F2FS_F_SB(folio)); > @@ -4057,9 +4050,9 @@ bool f2fs_alloc_nid(struct f2fs_sb_info *sbi, nid_t *nid); > void f2fs_alloc_nid_done(struct f2fs_sb_info *sbi, nid_t nid); > void f2fs_alloc_nid_failed(struct f2fs_sb_info *sbi, nid_t nid); > int f2fs_try_to_free_nids(struct f2fs_sb_info *sbi, int nr_shrink); > -int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio); > -int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio); > -int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct folio *folio); > +int f2fs_recover_inline_xattr(struct inode *inode, struct f2fs_cached_block *entry); > +int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry); > +int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry); > int f2fs_restore_node_summary(struct f2fs_sb_info *sbi, > unsigned int segno, struct f2fs_summary_block *sum); > int f2fs_flush_nat_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc); > @@ -4109,11 +4102,13 @@ int f2fs_allocate_new_segments(struct f2fs_sb_info *sbi); > int f2fs_trim_fs(struct f2fs_sb_info *sbi, struct fstrim_range *range); > bool f2fs_exist_trim_candidates(struct f2fs_sb_info *sbi, > struct cp_control *cpc); > -struct folio *f2fs_get_sum_folio(struct f2fs_sb_info *sbi, unsigned int segno); > +struct f2fs_cached_block *f2fs_get_sum_cache(struct f2fs_sb_info *sbi, > + unsigned int segno); > void f2fs_update_meta_page(struct f2fs_sb_info *sbi, void *src, > block_t blk_addr); > -void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct folio *folio, > - enum iostat_type io_type); > +void f2fs_do_write_meta_cache(struct f2fs_sb_info *sbi, > + struct f2fs_cached_block *entry, > + enum iostat_type io_type); > void f2fs_do_write_node_page(unsigned int nid, struct f2fs_io_info *fio); > void f2fs_outplace_write_data(struct dnode_of_data *dn, > struct f2fs_io_info *fio); > @@ -4136,8 +4131,6 @@ void f2fs_update_device_state(struct f2fs_sb_info *sbi, nid_t ino, > block_t blkaddr, unsigned int blkcnt); > void f2fs_folio_wait_writeback(struct folio *folio, enum page_type type, > bool ordered, bool locked); > -#define f2fs_wait_on_page_writeback(page, type, ordered, locked) \ > - f2fs_folio_wait_writeback(page_folio(page), type, ordered, locked) > void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr); > void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr, > block_t len); > @@ -4203,20 +4196,21 @@ void f2fs_unlock_op(struct f2fs_sb_info *sbi, struct f2fs_lock_context *lc); > void f2fs_stop_checkpoint(struct f2fs_sb_info *sbi, bool end_io, > unsigned char reason); > void f2fs_flush_ckpt_thread(struct f2fs_sb_info *sbi); > -struct folio *f2fs_grab_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index); > -struct folio *f2fs_get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index); > -struct folio *f2fs_get_meta_folio_retry(struct f2fs_sb_info *sbi, pgoff_t index); > -struct folio *f2fs_get_tmp_folio(struct f2fs_sb_info *sbi, pgoff_t index); > +struct f2fs_cached_block *f2fs_grab_meta_cache(struct f2fs_sb_info *sbi, pgoff_t index); > +struct f2fs_cached_block *f2fs_get_meta_cache(struct f2fs_sb_info *sbi, pgoff_t index); > +struct f2fs_cached_block *f2fs_get_meta_cache_retry(struct f2fs_sb_info *sbi, pgoff_t index); > +struct f2fs_cached_block *f2fs_get_tmp_cache(struct f2fs_sb_info *sbi, pgoff_t index); > bool f2fs_is_valid_blkaddr(struct f2fs_sb_info *sbi, > block_t blkaddr, int type); > bool f2fs_is_valid_blkaddr_raw(struct f2fs_sb_info *sbi, > block_t blkaddr, int type); > -int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, > +int f2fs_ra_meta_caches(struct f2fs_sb_info *sbi, block_t start, int nrpages, > int type, bool sync); > -void f2fs_ra_meta_pages_cond(struct f2fs_sb_info *sbi, pgoff_t index, > +void f2fs_ra_meta_caches_cond(struct f2fs_sb_info *sbi, pgoff_t index, > unsigned int ra_blocks); > -long f2fs_sync_meta_pages(struct f2fs_sb_info *sbi, long nr_to_write, > - enum iostat_type io_type); > +void f2fs_write_meta_caches(struct f2fs_sb_info *sbi); > +long f2fs_sync_meta_caches(struct f2fs_sb_info *sbi, long nr_to_write, > + bool sync, enum iostat_type io_type); > void f2fs_add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type); > void f2fs_remove_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type); > void f2fs_release_ino_entry(struct f2fs_sb_info *sbi, bool all); > @@ -4403,7 +4397,7 @@ struct f2fs_stat_info { > unsigned int bimodal, avg_vblocks; > int util_free, util_valid, util_invalid; > int rsvd_segs, overp_segs; > - int dirty_count, node_pages, meta_pages, compress_pages; > + int dirty_count, node_pages, meta_caches, compress_pages; > int compress_page_hit; > int prefree_count, free_segs, free_secs; > int cp_call_count[MAX_CALL_TYPE], cp_count; > @@ -4605,7 +4599,6 @@ extern const struct file_operations f2fs_file_operations; > extern const struct inode_operations f2fs_file_inode_operations; > extern const struct address_space_operations f2fs_dblock_aops; > extern const struct address_space_operations f2fs_node_aops; > -extern const struct address_space_operations f2fs_meta_aops; > extern const struct inode_operations f2fs_dir_inode_operations; > extern const struct inode_operations f2fs_symlink_inode_operations; > extern const struct inode_operations f2fs_encrypted_symlink_inode_operations; > @@ -4626,7 +4619,7 @@ int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio); > int f2fs_convert_inline_inode(struct inode *inode); > int f2fs_try_convert_inline_dir(struct inode *dir, struct dentry *dentry); > int f2fs_write_inline_data(struct inode *inode, struct folio *folio); > -int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio); > +int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entry); > struct f2fs_dir_entry *f2fs_find_in_inline_dir(struct inode *dir, > const struct f2fs_filename *fname, struct folio **res_folio, > bool use_hash); > @@ -5186,10 +5179,8 @@ static inline void f2fs_schedule_timeout_killable(long timeout, bool io) > } > > static inline void f2fs_handle_page_eio(struct f2fs_sb_info *sbi, > - struct folio *folio, enum page_type type) > + pgoff_t ofs, enum page_type type) > { > - pgoff_t ofs = folio->index; > - > if (unlikely(f2fs_cp_error(sbi))) > return; > > @@ -5224,36 +5215,10 @@ static inline bool f2fs_is_readonly(struct f2fs_sb_info *sbi) > return f2fs_sb_has_readonly(sbi) || f2fs_readonly(sbi->sb); > } > > -static inline void f2fs_truncate_meta_inode_pages(struct f2fs_sb_info *sbi, > - block_t blkaddr, unsigned int cnt) > -{ > - bool need_submit = false; > - int i = 0; > - > - do { > - struct folio *folio; > - > - folio = filemap_get_folio(META_MAPPING(sbi), blkaddr + i); > - if (!IS_ERR(folio)) { > - if (folio_test_writeback(folio)) > - need_submit = true; > - f2fs_folio_put(folio, false); > - } > - } while (++i < cnt && !need_submit); > - > - if (need_submit) > - f2fs_submit_merged_write_cond(sbi, sbi->meta_inode, > - NULL, 0, DATA); > - > - truncate_inode_pages_range(META_MAPPING(sbi), > - F2FS_BLK_TO_BYTES((loff_t)blkaddr), > - F2FS_BLK_END_BYTES((loff_t)(blkaddr + cnt - 1))); > -} > - > static inline void f2fs_invalidate_internal_cache(struct f2fs_sb_info *sbi, > block_t blkaddr, unsigned int len) > { > - f2fs_truncate_meta_inode_pages(sbi, blkaddr, len); > + f2fs_truncate_meta_caches(sbi, blkaddr, len); > f2fs_invalidate_compress_pages_range(sbi, blkaddr, len); > } > > diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c > index a54b3ab52f1a..92daa41dd96d 100644 > --- a/fs/f2fs/file.c > +++ b/fs/f2fs/file.c > @@ -214,7 +214,7 @@ static vm_fault_t f2fs_vm_page_mkwrite(struct vm_fault *vmf) > > f2fs_folio_wait_writeback(folio, DATA, false, true); > > - /* wait for GCed page writeback via META_MAPPING */ > + /* wait for GCed page writeback via generic cache */ > f2fs_wait_on_block_writeback(inode, dn.data_blkaddr); > > /* > @@ -2563,7 +2563,7 @@ int f2fs_do_shutdown(struct f2fs_sb_info *sbi, unsigned int flag, > f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_SHUTDOWN); > break; > case F2FS_GOING_DOWN_METAFLUSH: > - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_META_IO); > + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_META_IO); > f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_SHUTDOWN); > break; > case F2FS_GOING_DOWN_NEED_FSCK: > diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c > index c4da2f31805b..54327cb2e27e 100644 > --- a/fs/f2fs/gc.c > +++ b/fs/f2fs/gc.c > @@ -1066,7 +1066,7 @@ static int gc_node_segment(struct f2fs_sb_info *sbi, > continue; > > if (phase == 0) { > - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), 1, > + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nid), 1, > META_NAT, true); > continue; > } > @@ -1217,7 +1217,8 @@ static int ra_data_block(struct inode *inode, pgoff_t index) > struct address_space *mapping = inode->i_mapping; > struct inode *atomic_inode = NULL; > struct dnode_of_data dn; > - struct folio *folio, *efolio; > + struct folio *folio; > + struct f2fs_cached_block *entry; > struct f2fs_io_info fio = { > .sbi = sbi, > .ino = inode->i_ino, > @@ -1227,6 +1228,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index) > .op_flags = 0, > .encrypted_page = NULL, > .in_list = 0, > + .is_cache = 1, > }; > int err = 0; > > @@ -1285,22 +1287,21 @@ static int ra_data_block(struct inode *inode, pgoff_t index) > > f2fs_wait_on_block_writeback(inode, dn.data_blkaddr); > > - efolio = f2fs_filemap_get_folio(META_MAPPING(sbi), dn.data_blkaddr, > - FGP_LOCK | FGP_CREAT, GFP_NOFS); > - if (IS_ERR(efolio)) { > - err = PTR_ERR(efolio); > + entry = f2fs_grab_cache(META_CACHE(sbi), dn.data_blkaddr, > + F2FS_CACHE_LOCK_CREATE); > + if (IS_ERR(entry)) { > + err = PTR_ERR(entry); > goto put_folio; > } > > - fio.encrypted_page = &efolio->page; > - > - if (folio_test_uptodate(efolio)) > - goto put_encrypted_page; > + if (f2fs_cache_test_uptodate(entry)) > + goto put_cache; > > - err = f2fs_submit_page_bio(&fio); > + fio.cache_entry = entry; > + err = f2fs_submit_cache_read(&fio); > if (err) > - goto put_encrypted_page; > - f2fs_put_page(fio.encrypted_page, false); > + goto put_cache; > + f2fs_put_cache(entry, false); > f2fs_folio_put(folio, true); > > f2fs_update_iostat(sbi, inode, FS_DATA_READ_IO, F2FS_BLKSIZE); > @@ -1309,8 +1310,8 @@ static int ra_data_block(struct inode *inode, pgoff_t index) > if (atomic_inode) > iput(atomic_inode); > return 0; > -put_encrypted_page: > - f2fs_put_page(fio.encrypted_page, true); > +put_cache: > + f2fs_put_cache(entry, true); > put_folio: > f2fs_folio_put(folio, true); > out_iput: > @@ -1320,7 +1321,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index) > } > > /* > - * Move data block via META_MAPPING while keeping locked data page. > + * Move data block via meta cache while keeping locked data page. > * This can be used to move blocks, aka LBAs, directly on disk. > */ > static int move_data_block(struct inode *inode, block_t bidx, > @@ -1337,11 +1338,13 @@ static int move_data_block(struct inode *inode, block_t bidx, > .op_flags = 0, > .encrypted_page = NULL, > .in_list = 0, > + .is_cache = 1, > }; > struct dnode_of_data dn; > struct f2fs_summary sum; > struct node_info ni; > - struct folio *folio, *mfolio, *efolio; > + struct folio *folio; > + struct f2fs_cached_block *sentry, *tentry; > block_t newaddr; > int err = 0; > bool lfs_mode = f2fs_lfs_mode(fio.sbi); > @@ -1406,20 +1409,20 @@ static int move_data_block(struct inode *inode, block_t bidx, > if (lfs_mode) > f2fs_down_write(&fio.sbi->io_order_lock); > > - mfolio = f2fs_grab_cache_folio(META_MAPPING(fio.sbi), > - fio.old_blkaddr, false); > - if (IS_ERR(mfolio)) { > - err = PTR_ERR(mfolio); > + sentry = f2fs_grab_cache(META_CACHE(fio.sbi), fio.old_blkaddr, > + F2FS_CACHE_LOCK_CREATE); > + if (IS_ERR(sentry)) { > + err = PTR_ERR(sentry); > goto up_out; > } > > - fio.encrypted_page = folio_file_page(mfolio, fio.old_blkaddr); > + fio.cache_entry = sentry; > > - /* read source block in mfolio */ > - if (!folio_test_uptodate(mfolio)) { > - err = f2fs_submit_page_bio(&fio); > + /* read source block in cache */ > + if (!f2fs_cache_test_uptodate(sentry)) { > + err = f2fs_submit_cache_read(&fio); > if (err) { > - f2fs_folio_put(mfolio, true); > + f2fs_put_cache(sentry, true); > goto up_out; > } > > @@ -1428,11 +1431,11 @@ static int move_data_block(struct inode *inode, block_t bidx, > f2fs_update_iostat(fio.sbi, NULL, FS_GDATA_READ_IO, > F2FS_BLKSIZE); > > - folio_lock(mfolio); > - if (unlikely(!is_meta_folio(mfolio) || > - !folio_test_uptodate(mfolio))) { > + f2fs_lock_cache(sentry); > + if (unlikely(!f2fs_is_meta_cache(sentry) || > + !f2fs_cache_test_uptodate(sentry))) { > err = -EIO; > - f2fs_folio_put(mfolio, true); > + f2fs_put_cache(sentry, true); > goto up_out; > } > } > @@ -1443,46 +1446,45 @@ static int move_data_block(struct inode *inode, block_t bidx, > err = f2fs_allocate_data_block(fio.sbi, NULL, fio.old_blkaddr, &newaddr, > &sum, type, NULL); > if (err) { > - f2fs_folio_put(mfolio, true); > + f2fs_put_cache(sentry, true); > /* filesystem should shutdown, no need to recovery block */ > goto up_out; > } > > - efolio = f2fs_filemap_get_folio(META_MAPPING(fio.sbi), newaddr, > - FGP_LOCK | FGP_CREAT, GFP_NOFS); > - if (IS_ERR(efolio)) { > - err = PTR_ERR(efolio); > - f2fs_folio_put(mfolio, true); > + tentry = f2fs_grab_cache(META_CACHE(fio.sbi), newaddr, > + F2FS_CACHE_LOCK_CREATE); > + if (IS_ERR(tentry)) { > + err = PTR_ERR(tentry); > + f2fs_put_cache(sentry, true); > goto recover_block; > } > > - fio.encrypted_page = &efolio->page; > + fio.cache_entry = tentry; > > /* write target block */ > - f2fs_wait_on_page_writeback(fio.encrypted_page, DATA, true, true); > - memcpy(page_address(fio.encrypted_page), > - folio_address(mfolio), PAGE_SIZE); > - f2fs_folio_put(mfolio, true); > + f2fs_cache_wait_writeback_cond(tentry, DATA); > + memcpy(cache_address(tentry), cache_address(sentry), PAGE_SIZE); > + f2fs_put_cache(sentry, true); > > f2fs_invalidate_internal_cache(fio.sbi, fio.old_blkaddr, 1); > > - set_page_dirty(fio.encrypted_page); > - if (clear_page_dirty_for_io(fio.encrypted_page)) > + f2fs_mark_cache_dirty(tentry); > + if (f2fs_clear_cache_dirty(tentry)) > dec_page_count(fio.sbi, F2FS_DIRTY_META); > > - set_page_writeback(fio.encrypted_page); > + f2fs_start_cache_writeback(tentry); > > fio.op = REQ_OP_WRITE; > fio.op_flags = REQ_SYNC; > fio.new_blkaddr = newaddr; > - f2fs_submit_page_write(&fio); > + f2fs_submit_cache_write(&fio); > > f2fs_update_iostat(fio.sbi, NULL, FS_GC_DATA_IO, F2FS_BLKSIZE); > > f2fs_update_data_blkaddr(&dn, newaddr); > set_inode_flag(inode, FI_APPEND_WRITE); > > - f2fs_put_page(fio.encrypted_page, true); > + f2fs_put_cache(tentry, true); > recover_block: > if (err) > f2fs_do_replace_block(fio.sbi, &sum, newaddr, fio.old_blkaddr, > @@ -1614,7 +1616,7 @@ static int gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum, > continue; > > if (phase == 0) { > - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), 1, > + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nid), 1, > META_NAT, true); > continue; > } > @@ -1818,28 +1820,31 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, > sum_blk_cnt = DIV_ROUND_UP(end_segno - segno, sbi->sums_per_block); > /* readahead multi ssa blocks those have contiguous address */ > if (__is_large_section(sbi)) > - f2fs_ra_meta_pages(sbi, GET_SUM_BLOCK(sbi, segno), > + f2fs_ra_meta_caches(sbi, GET_SUM_BLOCK(sbi, segno), > sum_blk_cnt, META_SSA, true); > > /* reference all summary page */ > while (segno < end_segno) { > - struct folio *sum_folio = f2fs_get_sum_folio(sbi, segno); > + struct f2fs_cached_block *sum_entry = > + f2fs_get_sum_cache(sbi, segno); > > segno += sbi->sums_per_block; > - if (IS_ERR(sum_folio)) { > - int err = PTR_ERR(sum_folio); > + if (IS_ERR(sum_entry)) { > + int err = PTR_ERR(sum_entry); > > end_segno = segno - sbi->sums_per_block; > segno = rounddown(start_segno, sbi->sums_per_block); > while (segno < end_segno) { > - sum_folio = filemap_get_folio(META_MAPPING(sbi), > + sum_entry = f2fs_find_meta_cache(sbi, > GET_SUM_BLOCK(sbi, segno)); > - folio_put_refs(sum_folio, 2); > + f2fs_put_cache(sum_entry, false); > + f2fs_put_cache(sum_entry, false); > segno += sbi->sums_per_block; > } > return err; > } > - folio_unlock(sum_folio); > + f2fs_unlock_cache(sum_entry); > + > } > > blk_start_plug(&plug); > @@ -1847,11 +1852,16 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, > segno = start_segno; > while (segno < end_segno) { > unsigned int cur_segno; > + unsigned int block_end_segno; > > /* find segment summary of victim */ > - struct folio *sum_folio = filemap_get_folio(META_MAPPING(sbi), > + struct f2fs_cached_block *sum_entry = > + f2fs_find_meta_cache(sbi, > GET_SUM_BLOCK(sbi, segno)); > - unsigned int block_end_segno = rounddown(segno, sbi->sums_per_block) > + > + f2fs_bug_on(sbi, IS_ERR(sum_entry)); > + > + block_end_segno = rounddown(segno, sbi->sums_per_block) > + sbi->sums_per_block; > > if (block_end_segno > end_segno) > @@ -1864,8 +1874,8 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, > goto next_block; > } > > - if (!folio_test_uptodate(sum_folio) || > - unlikely(f2fs_cp_error(sbi))) > + if (!f2fs_cache_test_uptodate(sum_entry) || > + unlikely(f2fs_cp_error(sbi))) > goto next_block; > > for (cur_segno = segno; cur_segno < block_end_segno; > @@ -1884,7 +1894,7 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, > data_type = (type == SUM_TYPE_DATA) ? DATA : NODE; > } > > - sum = SUM_BLK_PAGE_ADDR(sbi, sum_folio, cur_segno); > + sum = SUM_BLK_ENTRY_ADDR(sbi, sum_entry, cur_segno); > if (type != GET_SUM_TYPE(sum_footer(sbi, sum))) { > f2fs_err(sbi, "Inconsistent segment (%u) type " > "[%d, %d] in SIT and SSA", > @@ -1926,12 +1936,14 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, > cur_segno + 1 : NULL_SEGNO; > > if (unlikely(freezing(current))) { > - folio_put_refs(sum_folio, 2); > + f2fs_put_cache(sum_entry, false); > + f2fs_put_cache(sum_entry, false); > goto stop; > } > } > next_block: > - folio_put_refs(sum_folio, 2); > + f2fs_put_cache(sum_entry, false); > + f2fs_put_cache(sum_entry, false); > segno = block_end_segno; > } > > diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c > index aec06fb4fd76..2156fb1fc57d 100644 > --- a/fs/f2fs/inline.c > +++ b/fs/f2fs/inline.c > @@ -294,7 +294,7 @@ int f2fs_write_inline_data(struct inode *inode, struct folio *folio) > return 0; > } > > -int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio) > +int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entry) > { > struct f2fs_sb_info *sbi = F2FS_I_SB(inode); > struct f2fs_inode *ri = NULL; > @@ -308,12 +308,13 @@ int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio) > * x o -> remove data blocks, and then recover inline_data > * x x -> recover data blocks > */ > - if (IS_INODE(nfolio)) > - ri = F2FS_INODE(nfolio); > + if (IS_INODE(cache_folio(entry))) > + ri = &CACHED_NODE(entry)->i; > > if (f2fs_has_inline_data(inode) && > ri && (ri->i_inline & F2FS_INLINE_DATA)) { > struct folio *ifolio; > + > process_inline: > ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); > if (IS_ERR(ifolio)) > @@ -321,7 +322,7 @@ int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio) > > f2fs_folio_wait_writeback(ifolio, NODE, true, true); > > - src_addr = inline_data_addr(inode, nfolio); > + src_addr = inline_data_addr(inode, cache_folio(entry)); > dst_addr = inline_data_addr(inode, ifolio); > memcpy(dst_addr, src_addr, MAX_INLINE_DATA(inode)); > > diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c > index bac1e360d966..c533da4d4d70 100644 > --- a/fs/f2fs/inode.c > +++ b/fs/f2fs/inode.c > @@ -577,7 +577,7 @@ static int do_read_inode(struct inode *inode) > > static bool is_meta_ino(struct f2fs_sb_info *sbi, unsigned int ino) > { > - if (ino == F2FS_NODE_INO(sbi) || ino == F2FS_META_INO(sbi)) > + if (ino == F2FS_NODE_INO(sbi)) > return true; > #ifdef CONFIG_F2FS_FS_COMPRESSION > if (test_opt(sbi, COMPRESS_CACHE) && ino == F2FS_COMPRESS_INO(sbi)) > @@ -624,9 +624,6 @@ struct inode *f2fs_iget(struct super_block *sb, unsigned long ino) > if (ino == F2FS_NODE_INO(sbi)) { > inode->i_mapping->a_ops = &f2fs_node_aops; > mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS); > - } else if (ino == F2FS_META_INO(sbi)) { > - inode->i_mapping->a_ops = &f2fs_meta_aops; > - mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS); > } else if (ino == F2FS_COMPRESS_INO(sbi)) { > #ifdef CONFIG_F2FS_FS_COMPRESSION > inode->i_mapping->a_ops = &f2fs_compress_aops; > @@ -824,8 +821,7 @@ int f2fs_write_inode(struct inode *inode, struct writeback_control *wbc) > { > struct f2fs_sb_info *sbi = F2FS_I_SB(inode); > > - if (inode->i_ino == F2FS_NODE_INO(sbi) || > - inode->i_ino == F2FS_META_INO(sbi)) > + if (inode->i_ino == F2FS_NODE_INO(sbi)) > return 0; > > /* > @@ -919,7 +915,6 @@ static bool f2fs_pre_evict_inode(struct inode *inode) > f2fs_invalidate_compress_pages(sbi, inode->i_ino); > > if (inode->i_ino == F2FS_NODE_INO(sbi) || > - inode->i_ino == F2FS_META_INO(sbi) || > inode->i_ino == F2FS_COMPRESS_INO(sbi)) > return true; > > diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c > index 46bea52e35c3..398c58fc6cad 100644 > --- a/fs/f2fs/node.c > +++ b/fs/f2fs/node.c > @@ -140,38 +140,36 @@ static void clear_node_folio_dirty(struct folio *folio) > folio_clear_uptodate(folio); > } > > -static struct folio *get_current_nat_folio(struct f2fs_sb_info *sbi, nid_t nid) > +static struct f2fs_cached_block *get_current_nat_cache(struct f2fs_sb_info *sbi, > + nid_t nid) > { > - return f2fs_get_meta_folio_retry(sbi, current_nat_addr(sbi, nid)); > + return f2fs_get_meta_cache_retry(sbi, current_nat_addr(sbi, nid)); > } > > -static struct folio *get_next_nat_folio(struct f2fs_sb_info *sbi, nid_t nid) > +static struct f2fs_cached_block *get_next_nat_cache(struct f2fs_sb_info *sbi, > + nid_t nid) > { > - struct folio *src_folio; > - struct folio *dst_folio; > + struct f2fs_cached_block *src_entry; > + struct f2fs_cached_block *dst_entry; > pgoff_t dst_off; > - void *src_addr; > - void *dst_addr; > struct f2fs_nm_info *nm_i = NM_I(sbi); > > dst_off = next_nat_addr(sbi, current_nat_addr(sbi, nid)); > > /* get current nat block page with lock */ > - src_folio = get_current_nat_folio(sbi, nid); > - if (IS_ERR(src_folio)) > - return src_folio; > - dst_folio = f2fs_grab_meta_folio(sbi, dst_off); > - f2fs_bug_on(sbi, folio_test_dirty(src_folio)); > - > - src_addr = folio_address(src_folio); > - dst_addr = folio_address(dst_folio); > - memcpy(dst_addr, src_addr, PAGE_SIZE); > - folio_mark_dirty(dst_folio); > - f2fs_folio_put(src_folio, true); > + src_entry = get_current_nat_cache(sbi, nid); > + if (IS_ERR(src_entry)) > + return src_entry; > + dst_entry = f2fs_grab_meta_cache(sbi, dst_off); > + f2fs_bug_on(sbi, f2fs_cache_test_dirty(src_entry)); > + > + memcpy(cache_address(dst_entry), cache_address(src_entry), sbi->blocksize); > + f2fs_mark_cache_dirty(dst_entry); > + f2fs_put_cache(src_entry, true); > > set_to_next_nat(nm_i, nid); > > - return dst_folio; > + return dst_entry; > } > > static struct nat_entry *__alloc_nat_entry(struct f2fs_sb_info *sbi, > @@ -575,7 +573,7 @@ int f2fs_get_node_info(struct f2fs_sb_info *sbi, nid_t nid, > struct f2fs_journal *journal = curseg->journal; > nid_t start_nid = START_NID(nid); > struct f2fs_nat_block *nat_blk; > - struct folio *folio = NULL; > + struct f2fs_cached_block *entry = NULL; > struct f2fs_nat_entry ne; > struct nat_entry *e; > pgoff_t index; > @@ -629,14 +627,14 @@ int f2fs_get_node_info(struct f2fs_sb_info *sbi, nid_t nid, > index = current_nat_addr(sbi, nid); > f2fs_up_read(&nm_i->nat_tree_lock); > > - folio = f2fs_get_meta_folio(sbi, index); > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > + entry = f2fs_get_meta_cache(sbi, index); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > > - nat_blk = folio_address(folio); > + nat_blk = cache_address(entry); > ne = nat_blk->entries[nid - start_nid]; > node_info_from_raw_nat(ni, &ne); > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > sanity_check: > if (__is_valid_data_blkaddr(ni->blk_addr) && > !f2fs_is_valid_blkaddr(sbi, ni->blk_addr, > @@ -1622,7 +1620,7 @@ static struct folio *__get_node_folio(struct f2fs_sb_info *sbi, pgoff_t nid, > out_put_err: > /* ENOENT comes from read_node_folio which is not an error. */ > if (err != -ENOENT) > - f2fs_handle_page_eio(sbi, folio, NODE); > + f2fs_handle_page_eio(sbi, folio->index, NODE); > f2fs_folio_put(folio, true); > return ERR_PTR(err); > } > @@ -2598,6 +2596,7 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, > bool sync, bool mount) > { > struct f2fs_nm_info *nm_i = NM_I(sbi); > + struct f2fs_cached_block *entry = NULL; > int i = 0, ret; > nid_t nid = nm_i->next_scan_nid; > > @@ -2623,7 +2622,7 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, > } > > /* readahead nat pages to be scanned */ > - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), FREE_NID_PAGES, > + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nid), FREE_NID_PAGES, > META_NAT, true); > > f2fs_down_read(&nm_i->nat_tree_lock); > @@ -2631,14 +2630,14 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, > while (1) { > if (!test_bit_le(NAT_BLOCK_OFFSET(nid), > nm_i->nat_block_bitmap)) { > - struct folio *folio = get_current_nat_folio(sbi, nid); > + entry = get_current_nat_cache(sbi, nid); > > - if (IS_ERR(folio)) { > - ret = PTR_ERR(folio); > + if (IS_ERR(entry)) { > + ret = PTR_ERR(entry); > } else { > - ret = scan_nat_page(sbi, folio_address(folio), > + ret = scan_nat_page(sbi, cache_address(entry), > nid); > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > } > > if (ret) { > @@ -2671,7 +2670,7 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, > > f2fs_up_read(&nm_i->nat_tree_lock); > > - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nm_i->next_scan_nid), > + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nm_i->next_scan_nid), > nm_i->ra_nid_pages, META_NAT, false); > > return 0; > @@ -2826,7 +2825,7 @@ int f2fs_try_to_free_nids(struct f2fs_sb_info *sbi, int nr_shrink) > return nr - nr_shrink; > } > > -int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) > +int f2fs_recover_inline_xattr(struct inode *inode, struct f2fs_cached_block *entry) > { > void *src_addr, *dst_addr; > size_t inline_size; > @@ -2837,7 +2836,7 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) > if (IS_ERR(ifolio)) > return PTR_ERR(ifolio); > > - ri = F2FS_INODE(folio); > + ri = &CACHED_NODE(entry)->i; > if (ri->i_inline & F2FS_INLINE_XATTR) { > if (!f2fs_has_inline_xattr(inode)) { > set_inode_flag(inode, FI_INLINE_XATTR); > @@ -2852,7 +2851,7 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) > } > > dst_addr = inline_xattr_addr(inode, ifolio); > - src_addr = inline_xattr_addr(inode, folio); > + src_addr = inline_xattr_addr(inode, cache_folio(entry)); > inline_size = inline_xattr_size(inode); > > f2fs_folio_wait_writeback(ifolio, NODE, true, true); > @@ -2863,7 +2862,7 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) > return 0; > } > > -int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio) > +int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry) > { > struct f2fs_sb_info *sbi = F2FS_I_SB(inode); > nid_t prev_xnid = F2FS_I(inode)->i_xattr_nid; > @@ -2901,8 +2900,8 @@ int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio) > f2fs_update_inode_page(inode); > > /* 3: update and set xattr node page dirty */ > - if (folio) { > - memcpy(F2FS_NODE(xfolio), F2FS_NODE(folio), > + if (entry) { > + memcpy(F2FS_NODE(xfolio), CACHED_NODE(entry), > VALID_XATTR_BLOCK_SIZE); > folio_mark_dirty(xfolio); > } > @@ -2911,10 +2910,10 @@ int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio) > return 0; > } > > -int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct folio *folio) > +int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) > { > struct f2fs_inode *src, *dst; > - nid_t ino = ino_of_node(folio); > + nid_t ino = ino_of_node(cache_folio(entry)); > struct node_info old_ni, new_ni; > struct folio *ifolio; > int err; > @@ -2940,7 +2939,7 @@ int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct folio *folio) > fill_node_footer(ifolio, ino, ino, 0, true); > set_cold_node(ifolio, false); > > - src = F2FS_INODE(folio); > + src = &CACHED_NODE(entry)->i; > dst = F2FS_INODE(ifolio); > > memcpy(dst, src, offsetof(struct f2fs_inode, i_ext)); > @@ -2999,24 +2998,24 @@ int f2fs_restore_node_summary(struct f2fs_sb_info *sbi, > nrpages = bio_max_segs(last_offset - i); > > /* readahead node pages */ > - f2fs_ra_meta_pages(sbi, addr, nrpages, META_POR, true); > + f2fs_ra_meta_caches(sbi, addr, nrpages, META_POR, true); > > for (idx = addr; idx < addr + nrpages; idx++) { > - struct folio *folio = f2fs_get_tmp_folio(sbi, idx); > + struct f2fs_cached_block *entry = > + f2fs_get_tmp_cache(sbi, idx); > > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > > - rn = F2FS_NODE(folio); > + rn = CACHED_NODE(entry); > sum_entry->nid = rn->footer.nid; > sum_entry->version = 0; > sum_entry->ofs_in_node = 0; > sum_entry++; > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > } > > - invalidate_mapping_pages(META_MAPPING(sbi), addr, > - addr + nrpages); > + f2fs_truncate_meta_caches(sbi, addr, nrpages); > } > return 0; > } > @@ -3126,7 +3125,7 @@ static int __flush_nat_entry_set(struct f2fs_sb_info *sbi, > bool to_journal = true; > struct f2fs_nat_block *nat_blk; > struct nat_entry *ne, *cur; > - struct folio *folio = NULL; > + struct f2fs_cached_block *entry = NULL; > > /* > * there are two steps to flush nat entries: > @@ -3140,11 +3139,11 @@ static int __flush_nat_entry_set(struct f2fs_sb_info *sbi, > if (to_journal) { > down_write(&curseg->journal_rwsem); > } else { > - folio = get_next_nat_folio(sbi, start_nid); > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > + entry = get_next_nat_cache(sbi, start_nid); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > > - nat_blk = folio_address(folio); > + nat_blk = cache_address(entry); > f2fs_bug_on(sbi, !nat_blk); > } > > @@ -3181,7 +3180,7 @@ static int __flush_nat_entry_set(struct f2fs_sb_info *sbi, > up_write(&curseg->journal_rwsem); > } else { > __update_nat_bits(sbi, start_nid, nat_blk); > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > } > > /* Allow dirty nats by node block allocation in write_begin */ > @@ -3252,7 +3251,7 @@ int f2fs_flush_nat_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) > __has_cursum_space(sbi, journal, > entry_count, NAT_JOURNAL)) > continue; > - f2fs_ra_meta_pages(sbi, set->set, 1, META_NAT, true); > + f2fs_ra_meta_caches(sbi, set->set, 1, META_NAT, true); > } > /* flush dirty nats in nat entry set */ > list_for_each_entry_safe(set, tmp, &sets, set_list) { > @@ -3288,15 +3287,15 @@ static int __get_nat_bitmaps(struct f2fs_sb_info *sbi) > nat_bits_addr = __start_cp_addr(sbi) + BLKS_PER_SEG(sbi) - > nm_i->nat_bits_blocks; > for (i = 0; i < nm_i->nat_bits_blocks; i++) { > - struct folio *folio; > + struct f2fs_cached_block *entry; > > - folio = f2fs_get_meta_folio(sbi, nat_bits_addr++); > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > + entry = f2fs_get_meta_cache(sbi, nat_bits_addr++); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > > memcpy(nm_i->nat_bits + F2FS_BLK_TO_BYTES(i), > - folio_address(folio), F2FS_BLKSIZE); > - f2fs_folio_put(folio, true); > + cache_address(entry), F2FS_BLKSIZE); > + f2fs_put_cache(entry, true); > } > > cp_ver |= (cur_cp_crc(ckpt) << 32); > diff --git a/fs/f2fs/node.h b/fs/f2fs/node.h > index 5e114f352099..de8dcd5d4392 100644 > --- a/fs/f2fs/node.h > +++ b/fs/f2fs/node.h > @@ -311,9 +311,9 @@ static inline void fill_node_footer_blkaddr(struct folio *folio, block_t blkaddr > rn->footer.next_blkaddr = cpu_to_le32(blkaddr); > } > > -static inline bool is_recoverable_dnode(const struct folio *folio) > +static inline bool is_recoverable_dnode(struct f2fs_sb_info *sbi, const struct folio *folio) > { > - struct f2fs_checkpoint *ckpt = F2FS_CKPT(F2FS_F_SB(folio)); > + struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); > __u64 cp_ver = cur_cp_version(ckpt); > > /* Don't care crc part, if fsck.f2fs sets it. */ > diff --git a/fs/f2fs/recovery.c b/fs/f2fs/recovery.c > index aaa5227739c8..5cb56b4c1879 100644 > --- a/fs/f2fs/recovery.c > +++ b/fs/f2fs/recovery.c > @@ -182,33 +182,33 @@ static const char *recover_printable_name(struct inode *inode, > return raw->i_name; > } > > -static int recover_dentry(struct inode *inode, struct folio *ifolio, > +static int recover_dentry(struct inode *inode, struct f2fs_cached_block *entry, > struct list_head *dir_list) > { > - struct f2fs_inode *raw_inode = F2FS_INODE(ifolio); > + struct f2fs_inode *raw_inode = &CACHED_NODE(entry)->i; > nid_t pino = le32_to_cpu(raw_inode->i_pino); > struct f2fs_dir_entry *de; > struct f2fs_filename fname; > struct qstr usr_fname; > struct folio *folio; > struct inode *dir, *einode; > - struct fsync_inode_entry *entry; > + struct fsync_inode_entry *fsync_entry; > int err = 0; > const char *name; > int name_len; > > - entry = get_fsync_inode(dir_list, pino); > - if (!entry) { > - entry = add_fsync_inode(F2FS_I_SB(inode), dir_list, > + fsync_entry = get_fsync_inode(dir_list, pino); > + if (!fsync_entry) { > + fsync_entry = add_fsync_inode(F2FS_I_SB(inode), dir_list, > pino, false); > - if (IS_ERR(entry)) { > - dir = ERR_CAST(entry); > - err = PTR_ERR(entry); > + if (IS_ERR(fsync_entry)) { > + dir = ERR_CAST(fsync_entry); > + err = PTR_ERR(fsync_entry); > goto out; > } > } > > - dir = entry->inode; > + dir = fsync_entry->inode; > err = init_recovered_filename(dir, inode, raw_inode, &fname, &usr_fname); > if (err) > goto out; > @@ -256,14 +256,14 @@ static int recover_dentry(struct inode *inode, struct folio *ifolio, > out: > name = recover_printable_name(inode, raw_inode, &name_len); > f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, dir = %llu, err = %d", > - __func__, ino_of_node(ifolio), name_len, name, > + __func__, ino_of_node(cache_folio(entry)), name_len, name, > IS_ERR(dir) ? 0 : dir->i_ino, err); > return err; > } > > -static int recover_quota_data(struct inode *inode, struct folio *folio) > +static int recover_quota_data(struct inode *inode, struct f2fs_cached_block *entry) > { > - struct f2fs_inode *raw = F2FS_INODE(folio); > + struct f2fs_inode *raw = &CACHED_NODE(entry)->i; > struct iattr attr; > uid_t i_uid = le32_to_cpu(raw->i_uid); > gid_t i_gid = le32_to_cpu(raw->i_gid); > @@ -300,9 +300,9 @@ static void recover_inline_flags(struct inode *inode, struct f2fs_inode *ri) > clear_inode_flag(inode, FI_DATA_EXIST); > } > > -static int recover_inode(struct inode *inode, struct folio *folio) > +static int recover_inode(struct inode *inode, struct f2fs_cached_block *entry) > { > - struct f2fs_inode *raw = F2FS_INODE(folio); > + struct f2fs_inode *raw = &CACHED_NODE(entry)->i; > struct f2fs_inode_info *fi = F2FS_I(inode); > const char *name; > int name_len; > @@ -310,7 +310,7 @@ static int recover_inode(struct inode *inode, struct folio *folio) > > inode->i_mode = le16_to_cpu(raw->i_mode); > > - err = recover_quota_data(inode, folio); > + err = recover_quota_data(inode, entry); > if (err) > return err; > > @@ -357,7 +357,7 @@ static int recover_inode(struct inode *inode, struct folio *folio) > name = recover_printable_name(inode, raw, &name_len); > > f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, inline = %x", > - __func__, ino_of_node(folio), name_len, name, > + __func__, ino_of_node(cache_folio(entry)), name_len, name, > raw->i_inline); > return 0; > } > @@ -386,30 +386,30 @@ static int sanity_check_node_chain(struct f2fs_sb_info *sbi, block_t blkaddr, > return 0; > > for (i = 0; i < 2; i++) { > - struct folio *folio; > + struct f2fs_cached_block *entry; > > if (!f2fs_is_valid_blkaddr(sbi, *blkaddr_fast, META_POR)) { > *is_detecting = false; > return 0; > } > > - folio = f2fs_get_tmp_folio(sbi, *blkaddr_fast); > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > + entry = f2fs_get_tmp_cache(sbi, *blkaddr_fast); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > > - if (!is_recoverable_dnode(folio)) { > - f2fs_folio_put(folio, true); > + if (!is_recoverable_dnode(sbi, cache_folio(entry))) { > + f2fs_put_cache(entry, true); > *is_detecting = false; > return 0; > } > > ra_blocks = adjust_por_ra_blocks(sbi, ra_blocks, *blkaddr_fast, > - next_blkaddr_of_node(folio)); > + next_blkaddr_of_node(cache_folio(entry))); > > - *blkaddr_fast = next_blkaddr_of_node(folio); > - f2fs_folio_put(folio, true); > + *blkaddr_fast = next_blkaddr_of_node(cache_folio(entry)); > + f2fs_put_cache(entry, true); > > - f2fs_ra_meta_pages_cond(sbi, *blkaddr_fast, ra_blocks); > + f2fs_ra_meta_caches_cond(sbi, *blkaddr_fast, ra_blocks); > } > > if (*blkaddr_fast == blkaddr) { > @@ -434,45 +434,47 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head, > blkaddr_fast = blkaddr; > > while (1) { > - struct fsync_inode_entry *entry; > - struct folio *folio; > + struct fsync_inode_entry *fsync_entry; > + struct f2fs_cached_block *entry; > + struct f2fs_node *rn; > > if (!f2fs_is_valid_blkaddr(sbi, blkaddr, META_POR)) > return 0; > > - folio = f2fs_get_tmp_folio(sbi, blkaddr); > - if (IS_ERR(folio)) { > - err = PTR_ERR(folio); > + entry = f2fs_get_tmp_cache(sbi, blkaddr); > + if (IS_ERR(entry)) { > + err = PTR_ERR(entry); > break; > } > + rn = CACHED_NODE(entry); > > - if (!is_recoverable_dnode(folio)) { > - f2fs_folio_put(folio, true); > + if (!is_recoverable_dnode(sbi, cache_folio(entry))) { > + f2fs_put_cache(entry, true); > break; > } > > - if (!is_fsync_dnode(folio)) > + if (!is_fsync_dnode(cache_folio(entry))) > goto next; > > - entry = get_fsync_inode(head, ino_of_node(folio)); > - if (!entry) { > + fsync_entry = get_fsync_inode(head, ino_of_node(cache_folio(entry))); > + if (!fsync_entry) { > bool quota_inode = false; > > if (!check_only && > - IS_INODE(folio) && > - is_dent_dnode(folio)) { > - err = f2fs_recover_inode_page(sbi, folio); > + IS_INODE(cache_folio(entry)) && > + is_dent_dnode(cache_folio(entry))) { > + err = f2fs_recover_inode_page(sbi, entry); > if (err) { > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > break; > } > quota_inode = true; > } > > - entry = add_fsync_inode(sbi, head, ino_of_node(folio), > + fsync_entry = add_fsync_inode(sbi, head, ino_of_node(cache_folio(entry)), > quota_inode); > - if (IS_ERR(entry)) { > - err = PTR_ERR(entry); > + if (IS_ERR(fsync_entry)) { > + err = PTR_ERR(fsync_entry); > /* > * CP | dnode(F) | inode(DF) > * For this case, we should not give up now. > @@ -482,18 +484,18 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head, > *new_inode = true; > goto next; > } > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > break; > } > } > - entry->blkaddr = blkaddr; > + fsync_entry->blkaddr = blkaddr; > > - if (IS_INODE(folio) && is_dent_dnode(folio)) > - entry->last_dentry = blkaddr; > + if (IS_INODE(cache_folio(entry)) && is_dent_dnode(cache_folio(entry))) > + fsync_entry->last_dentry = blkaddr; > next: > /* check next segment */ > - blkaddr = next_blkaddr_of_node(folio); > - f2fs_folio_put(folio, true); > + blkaddr = next_blkaddr_of_node(cache_folio(entry)); > + f2fs_put_cache(entry, true); > > err = sanity_check_node_chain(sbi, blkaddr, &blkaddr_fast, > &is_detecting); > @@ -519,7 +521,8 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, > unsigned short blkoff = GET_BLKOFF_FROM_SEG0(sbi, blkaddr); > struct f2fs_summary_block *sum_node; > struct f2fs_summary sum; > - struct folio *sum_folio, *node_folio; > + struct f2fs_cached_block *entry = NULL; > + struct folio *node_folio; > struct dnode_of_data tdn = *dn; > nid_t ino, nid; > struct inode *inode; > @@ -541,12 +544,12 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, > } > } > > - sum_folio = f2fs_get_sum_folio(sbi, segno); > - if (IS_ERR(sum_folio)) > - return PTR_ERR(sum_folio); > - sum_node = SUM_BLK_PAGE_ADDR(sbi, sum_folio, segno); > + entry = f2fs_get_sum_cache(sbi, segno); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > + sum_node = SUM_BLK_ENTRY_ADDR(sbi, entry, segno); > sum = sum_entries(sum_node)[blkoff]; > - f2fs_folio_put(sum_folio, true); > + f2fs_put_cache(entry, true); > got_it: > /* Use the locked dnode page and inode */ > nid = le32_to_cpu(sum.nid); > @@ -645,7 +648,7 @@ static int f2fs_reserve_new_block_retry(struct dnode_of_data *dn) > } > > static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, > - struct folio *folio) > + struct f2fs_cached_block *entry) > { > struct dnode_of_data dn; > struct node_info ni; > @@ -653,19 +656,19 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, > int err = 0, recovered = 0; > > /* step 1: recover xattr */ > - if (IS_INODE(folio)) { > - err = f2fs_recover_inline_xattr(inode, folio); > + if (IS_INODE(cache_folio(entry))) { > + err = f2fs_recover_inline_xattr(inode, entry); > if (err) > goto out; > - } else if (f2fs_has_xattr_block(ofs_of_node(folio))) { > - err = f2fs_recover_xattr_data(inode, folio); > + } else if (f2fs_has_xattr_block(ofs_of_node(cache_folio(entry)))) { > + err = f2fs_recover_xattr_data(inode, entry); > if (!err) > recovered++; > goto out; > } > > /* step 2: recover inline data */ > - err = f2fs_recover_inline_data(inode, folio); > + err = f2fs_recover_inline_data(inode, entry); > if (err) { > if (err == 1) > err = 0; > @@ -673,8 +676,8 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, > } > > /* step 3: recover data indices */ > - start = f2fs_start_bidx_of_node(ofs_of_node(folio), inode); > - end = start + ADDRS_PER_PAGE(folio, inode); > + start = f2fs_start_bidx_of_node(ofs_of_node(cache_folio(entry)), inode); > + end = start + addrs_per_page(inode, IS_INODE(cache_folio(entry))); > > set_new_dnode(&dn, inode, NULL, NULL, 0); > retry_dn: > @@ -693,12 +696,12 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, > if (err) > goto err; > > - f2fs_bug_on(sbi, ni.ino != ino_of_node(folio)); > + f2fs_bug_on(sbi, ni.ino != ino_of_node(cache_folio(entry))); > > - if (ofs_of_node(dn.node_folio) != ofs_of_node(folio)) { > + if (ofs_of_node(dn.node_folio) != ofs_of_node(cache_folio(entry))) { > f2fs_warn(sbi, "Inconsistent ofs_of_node, ino:%llu, ofs:%u, %u", > inode->i_ino, ofs_of_node(dn.node_folio), > - ofs_of_node(folio)); > + ofs_of_node(cache_folio(entry))); > err = -EFSCORRUPTED; > f2fs_handle_error(sbi, ERROR_INCONSISTENT_FOOTER); > fserror_report_file_metadata(dn.inode, err, GFP_NOFS); > @@ -709,7 +712,7 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, > block_t src, dest; > > src = f2fs_data_blkaddr(&dn); > - dest = data_blkaddr(dn.inode, folio, dn.ofs_in_node); > + dest = data_blkaddr(dn.inode, cache_folio(entry), dn.ofs_in_node); > > if (__is_valid_data_blkaddr(src) && > !f2fs_is_valid_blkaddr(sbi, src, META_POR)) { > @@ -784,16 +787,16 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, > } > } > > - copy_node_footer(dn.node_folio, folio); > + copy_node_footer(dn.node_folio, cache_folio(entry)); > fill_node_footer(dn.node_folio, dn.nid, ni.ino, > - ofs_of_node(folio), false); > + ofs_of_node(cache_folio(entry)), false); > folio_mark_dirty(dn.node_folio); > err: > f2fs_put_dnode(&dn); > out: > f2fs_notice(sbi, "recover_data: ino = %llx, nid = %x (i_size: %s), " > "range (%u, %u), recovered = %d, err = %d", > - inode->i_ino, nid_of_node(folio), > + inode->i_ino, nid_of_node(cache_folio(entry)), > file_keep_isize(inode) ? "keep" : "recover", > start, end, recovered, err); > return err; > @@ -820,26 +823,26 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list, > blkaddr = NEXT_FREE_BLKADDR(sbi, curseg); > > while (1) { > - struct fsync_inode_entry *entry; > - struct folio *folio; > + struct fsync_inode_entry *fsync_entry; > + struct f2fs_cached_block *entry; > > if (!f2fs_is_valid_blkaddr(sbi, blkaddr, META_POR)) > break; > > - folio = f2fs_get_tmp_folio(sbi, blkaddr); > - if (IS_ERR(folio)) { > - err = PTR_ERR(folio); > + entry = f2fs_get_tmp_cache(sbi, blkaddr); > + if (IS_ERR(entry)) { > + err = PTR_ERR(entry); > break; > } > > - if (!is_recoverable_dnode(folio)) { > - f2fs_folio_put(folio, true); > + if (!is_recoverable_dnode(sbi, cache_folio(entry))) { > + f2fs_put_cache(entry, true); > break; > } > recoverable_dnode++; > > - entry = get_fsync_inode(inode_list, ino_of_node(folio)); > - if (!entry) > + fsync_entry = get_fsync_inode(inode_list, ino_of_node(cache_folio(entry))); > + if (!fsync_entry) > goto next; > fsynced_dnode++; > /* > @@ -847,40 +850,40 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list, > * In this case, we can lose the latest inode(x). > * So, call recover_inode for the inode update. > */ > - if (IS_INODE(folio)) { > - err = recover_inode(entry->inode, folio); > + if (IS_INODE(cache_folio(entry))) { > + err = recover_inode(fsync_entry->inode, entry); > if (err) { > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > break; > } > recovered_inode++; > } > - if (entry->last_dentry == blkaddr) { > - err = recover_dentry(entry->inode, folio, dir_list); > + if (fsync_entry->last_dentry == blkaddr) { > + err = recover_dentry(fsync_entry->inode, entry, dir_list); > if (err) { > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > break; > } > recovered_dentry++; > } > - err = do_recover_data(sbi, entry->inode, folio); > + err = do_recover_data(sbi, fsync_entry->inode, entry); > if (err) { > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > break; > } > recovered_dnode++; > > - if (entry->blkaddr == blkaddr) > - list_move_tail(&entry->list, tmp_inode_list); > + if (fsync_entry->blkaddr == blkaddr) > + list_move_tail(&fsync_entry->list, tmp_inode_list); > next: > ra_blocks = adjust_por_ra_blocks(sbi, ra_blocks, blkaddr, > - next_blkaddr_of_node(folio)); > + next_blkaddr_of_node(cache_folio(entry))); > > /* check next segment */ > - blkaddr = next_blkaddr_of_node(folio); > - f2fs_folio_put(folio, true); > + blkaddr = next_blkaddr_of_node(cache_folio(entry)); > + f2fs_put_cache(entry, true); > > - f2fs_ra_meta_pages_cond(sbi, blkaddr, ra_blocks); > + f2fs_ra_meta_caches_cond(sbi, blkaddr, ra_blocks); > total_dnode++; > } > if (!err) > @@ -937,12 +940,11 @@ int f2fs_recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only) > destroy_fsync_dnodes(&tmp_inode_list, err); > > /* truncate meta pages to be used by the recovery */ > - truncate_inode_pages_range(META_MAPPING(sbi), > - (loff_t)MAIN_BLKADDR(sbi) << PAGE_SHIFT, -1); > - > + f2fs_truncate_meta_caches(sbi, MAIN_BLKADDR(sbi), > + MAX_BLKADDR(sbi) - MAIN_BLKADDR(sbi)); > if (err) { > truncate_inode_pages_final(NODE_MAPPING(sbi)); > - truncate_inode_pages_final(META_MAPPING(sbi)); > + f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); > } > > /* > diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c > index 56decf9c691c..cbb3a8c9f4ab 100644 > --- a/fs/f2fs/segment.c > +++ b/fs/f2fs/segment.c > @@ -2774,63 +2774,62 @@ int f2fs_npages_for_summary_flush(struct f2fs_sb_info *sbi, bool for_ra) > return 3; > } > > -/* > - * Caller should put this summary folio > - */ > -struct folio *f2fs_get_sum_folio(struct f2fs_sb_info *sbi, unsigned int segno) > + > +struct f2fs_cached_block *f2fs_get_sum_cache(struct f2fs_sb_info *sbi, > + unsigned int segno) > { > if (unlikely(f2fs_cp_error(sbi))) > return ERR_PTR(-EIO); > - return f2fs_get_meta_folio_retry(sbi, GET_SUM_BLOCK(sbi, segno)); > + return f2fs_get_meta_cache_retry(sbi, GET_SUM_BLOCK(sbi, segno)); > } > > void f2fs_update_meta_page(struct f2fs_sb_info *sbi, > void *src, block_t blk_addr) > { > - struct folio *folio; > + struct f2fs_cached_block *entry; > > if (!f2fs_sb_has_packed_ssa(sbi)) > - folio = f2fs_grab_meta_folio(sbi, blk_addr); > + entry = f2fs_grab_meta_cache(sbi, blk_addr); > else > - folio = f2fs_get_meta_folio_retry(sbi, blk_addr); > + entry = f2fs_get_meta_cache_retry(sbi, blk_addr); > > - if (IS_ERR(folio)) > + if (IS_ERR(entry)) > return; > > - memcpy(folio_address(folio), src, PAGE_SIZE); > - folio_mark_dirty(folio); > - f2fs_folio_put(folio, true); > + memcpy(cache_address(entry), src, sbi->blocksize); > + f2fs_mark_cache_dirty(entry); > + f2fs_put_cache(entry, true); > } > > static void write_sum_page(struct f2fs_sb_info *sbi, > struct f2fs_summary_block *sum_blk, unsigned int segno) > { > - struct folio *folio; > + struct f2fs_cached_block *entry; > > if (!f2fs_sb_has_packed_ssa(sbi)) > return f2fs_update_meta_page(sbi, (void *)sum_blk, > GET_SUM_BLOCK(sbi, segno)); > > - folio = f2fs_get_sum_folio(sbi, segno); > - if (IS_ERR(folio)) > + entry = f2fs_get_sum_cache(sbi, GET_SUM_BLOCK(sbi, segno)); > + if (IS_ERR(entry)) > return; > > - memcpy(SUM_BLK_PAGE_ADDR(sbi, folio, segno), sum_blk, > + memcpy(SUM_BLK_ENTRY_ADDR(sbi, entry, segno), sum_blk, > sbi->sum_blocksize); > - folio_mark_dirty(folio); > - f2fs_folio_put(folio, true); > + f2fs_mark_cache_dirty(entry); > + f2fs_put_cache(entry, true); > } > > static void write_current_sum_page(struct f2fs_sb_info *sbi, > int type, block_t blk_addr) > { > struct curseg_info *curseg = CURSEG_I(sbi, type); > - struct folio *folio = f2fs_grab_meta_folio(sbi, blk_addr); > + struct f2fs_cached_block *entry = f2fs_grab_meta_cache(sbi, blk_addr); > struct f2fs_summary_block *src = curseg->sum_blk; > struct f2fs_summary_block *dst; > > - dst = folio_address(folio); > - memset(dst, 0, PAGE_SIZE); > + dst = cache_address(entry); > + memset(dst, 0, sbi->blocksize); > > mutex_lock(&curseg->curseg_mutex); > > @@ -2843,8 +2842,8 @@ static void write_current_sum_page(struct f2fs_sb_info *sbi, > > mutex_unlock(&curseg->curseg_mutex); > > - folio_mark_dirty(folio); > - f2fs_folio_put(folio, true); > + f2fs_mark_cache_dirty(entry); > + f2fs_put_cache(entry, true); > } > > static int is_next_segment_free(struct f2fs_sb_info *sbi, > @@ -3160,7 +3159,7 @@ static int change_curseg(struct f2fs_sb_info *sbi, int type) > struct curseg_info *curseg = CURSEG_I(sbi, type); > unsigned int new_segno = curseg->next_segno; > struct f2fs_summary_block *sum_node; > - struct folio *sum_folio; > + struct f2fs_cached_block *entry = NULL; > > if (curseg->inited) > write_sum_page(sbi, curseg->sum_blk, curseg->segno); > @@ -3176,15 +3175,15 @@ static int change_curseg(struct f2fs_sb_info *sbi, int type) > curseg->alloc_type = SSR; > curseg->next_blkoff = __next_free_blkoff(sbi, curseg->segno, 0); > > - sum_folio = f2fs_get_sum_folio(sbi, new_segno); > - if (IS_ERR(sum_folio)) { > + entry = f2fs_get_sum_cache(sbi, new_segno); > + if (IS_ERR(entry)) { > /* GC won't be able to use stale summary pages by cp_error */ > memset(curseg->sum_blk, 0, sbi->sum_entry_size); > - return PTR_ERR(sum_folio); > + return PTR_ERR(entry); > } > - sum_node = SUM_BLK_PAGE_ADDR(sbi, sum_folio, new_segno); > + sum_node = SUM_BLK_ENTRY_ADDR(sbi, entry, new_segno); > memcpy(curseg->sum_blk, sum_node, sbi->sum_entry_size); > - f2fs_folio_put(sum_folio, true); > + f2fs_put_cache(entry, true); > return 0; > } > > @@ -4127,8 +4126,9 @@ static void do_write_page(struct f2fs_summary *sum, struct f2fs_io_info *fio) > f2fs_up_read(&fio->sbi->io_order_lock); > } > > -void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct folio *folio, > - enum iostat_type io_type) > +void f2fs_do_write_meta_cache(struct f2fs_sb_info *sbi, > + struct f2fs_cached_block *entry, > + enum iostat_type io_type) > { > struct f2fs_io_info fio = { > .sbi = sbi, > @@ -4136,20 +4136,21 @@ void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct folio *folio, > .temp = HOT, > .op = REQ_OP_WRITE, > .op_flags = REQ_SYNC | REQ_META | REQ_PRIO, > - .old_blkaddr = folio->index, > - .new_blkaddr = folio->index, > - .folio = folio, > + .old_blkaddr = entry->index, > + .new_blkaddr = entry->index, > .encrypted_page = NULL, > .in_list = 0, > + .cache_entry = entry, > + .is_cache = 1, > }; > > - if (unlikely(folio->index >= MAIN_BLKADDR(sbi))) > + if (unlikely(entry->index >= MAIN_BLKADDR(sbi))) > fio.op_flags &= ~REQ_META; > > - folio_start_writeback(folio); > - f2fs_submit_page_write(&fio); > + f2fs_start_cache_writeback(entry); > + f2fs_submit_cache_write(&fio); > > - stat_inc_meta_count(sbi, folio->index); > + stat_inc_meta_count(sbi, entry->index); > f2fs_update_iostat(sbi, NULL, io_type, F2FS_BLKSIZE); > } > > @@ -4206,7 +4207,7 @@ int f2fs_inplace_write_data(struct f2fs_io_info *fio) > } > > if (fio->meta_gc) > - f2fs_truncate_meta_inode_pages(sbi, fio->new_blkaddr, 1); > + f2fs_truncate_meta_caches(sbi, fio->new_blkaddr, 1); > > stat_inc_inplace_blocks(fio->sbi); > > @@ -4372,7 +4373,7 @@ void f2fs_folio_wait_writeback(struct folio *folio, enum page_type type, > void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr) > { > struct f2fs_sb_info *sbi = F2FS_I_SB(inode); > - struct folio *cfolio; > + struct f2fs_cached_block *entry; > > if (!f2fs_meta_inode_gc_required(inode)) > return; > @@ -4380,11 +4381,12 @@ void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr) > if (!__is_valid_data_blkaddr(blkaddr)) > return; > > - cfolio = filemap_lock_folio(META_MAPPING(sbi), blkaddr); > - if (!IS_ERR(cfolio)) { > - f2fs_folio_wait_writeback(cfolio, DATA, true, true); > - f2fs_folio_put(cfolio, true); > - } > + entry = f2fs_find_cache(META_CACHE(sbi), blkaddr); > + if (IS_ERR(entry)) > + return; > + f2fs_lock_cache(entry); > + f2fs_cache_wait_writeback_cond(entry, DATA); > + f2fs_put_cache(entry, true); > } > > void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr, > @@ -4399,7 +4401,7 @@ void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr, > for (i = 0; i < len; i++) > f2fs_wait_on_block_writeback(inode, blkaddr + i); > > - f2fs_truncate_meta_inode_pages(sbi, blkaddr, len); > + f2fs_truncate_meta_caches(sbi, blkaddr, len); > } > > static int read_compacted_summaries(struct f2fs_sb_info *sbi) > @@ -4407,16 +4409,16 @@ static int read_compacted_summaries(struct f2fs_sb_info *sbi) > struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); > struct curseg_info *seg_i; > unsigned char *kaddr; > - struct folio *folio; > + struct f2fs_cached_block *entry; > block_t start; > int i, j, offset; > > start = start_sum_block(sbi); > > - folio = f2fs_get_meta_folio(sbi, start++); > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > - kaddr = folio_address(folio); > + entry = f2fs_get_meta_cache(sbi, start++); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > + kaddr = cache_address(entry); > > /* Step 1: restore nat cache */ > seg_i = CURSEG_I(sbi, CURSEG_HOT_DATA); > @@ -4453,16 +4455,16 @@ static int read_compacted_summaries(struct f2fs_sb_info *sbi) > SUM_FOOTER_SIZE) > continue; > > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > > - folio = f2fs_get_meta_folio(sbi, start++); > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > - kaddr = folio_address(folio); > + entry = f2fs_get_meta_cache(sbi, start++); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > + kaddr = cache_address(entry); > offset = 0; > } > } > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > return 0; > } > > @@ -4471,7 +4473,7 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type) > struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); > struct f2fs_summary_block *sum; > struct curseg_info *curseg; > - struct folio *new; > + struct f2fs_cached_block *entry; > unsigned short blk_off; > unsigned int segno = 0; > block_t blk_addr = 0; > @@ -4498,10 +4500,10 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type) > blk_addr = GET_SUM_BLOCK(sbi, segno); > } > > - new = f2fs_get_meta_folio(sbi, blk_addr); > - if (IS_ERR(new)) > - return PTR_ERR(new); > - sum = folio_address(new); > + entry = f2fs_get_meta_cache(sbi, blk_addr); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > + sum = cache_address(entry); > > if (IS_NODESEG(type)) { > if (__exist_node_summaries(sbi)) { > @@ -4538,7 +4540,7 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type) > curseg->next_blkoff = blk_off; > mutex_unlock(&curseg->curseg_mutex); > out: > - f2fs_folio_put(new, true); > + f2fs_put_cache(entry, true); > return err; > } > > @@ -4553,8 +4555,8 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi) > int npages = f2fs_npages_for_summary_flush(sbi, true); > > if (npages >= 2) > - f2fs_ra_meta_pages(sbi, start_sum_block(sbi), npages, > - META_CP, true); > + f2fs_ra_meta_caches(sbi, start_sum_block(sbi), > + npages, META_CP, true); > > /* restore for compacted data summary */ > err = read_compacted_summaries(sbi); > @@ -4564,7 +4566,7 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi) > } > > if (__exist_node_summaries(sbi)) > - f2fs_ra_meta_pages(sbi, > + f2fs_ra_meta_caches(sbi, > sum_blk_addr(sbi, NR_CURSEG_PERSIST_TYPE, type), > NR_CURSEG_PERSIST_TYPE - type, META_CP, true); > > @@ -4587,16 +4589,16 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi) > > static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr) > { > - struct folio *folio; > + struct f2fs_cached_block *entry = NULL; > unsigned char *kaddr; > struct f2fs_summary *summary; > struct curseg_info *seg_i; > int written_size = 0; > int i, j; > > - folio = f2fs_grab_meta_folio(sbi, blkaddr++); > - kaddr = folio_address(folio); > - memset(kaddr, 0, PAGE_SIZE); > + entry = f2fs_grab_meta_cache(sbi, blkaddr++); > + kaddr = cache_address(entry); > + memset(kaddr, 0, sbi->blocksize); > > /* Step 1: write nat cache */ > seg_i = CURSEG_I(sbi, CURSEG_HOT_DATA); > @@ -4612,10 +4614,10 @@ static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr) > for (i = CURSEG_HOT_DATA; i <= CURSEG_COLD_DATA; i++) { > seg_i = CURSEG_I(sbi, i); > for (j = 0; j < f2fs_curseg_valid_blocks(sbi, i); j++) { > - if (!folio) { > - folio = f2fs_grab_meta_folio(sbi, blkaddr++); > - kaddr = folio_address(folio); > - memset(kaddr, 0, PAGE_SIZE); > + if (!entry) { > + entry = f2fs_grab_meta_cache(sbi, blkaddr++); > + kaddr = cache_address(entry); > + memset(kaddr, 0, sbi->blocksize); > written_size = 0; > } > summary = (struct f2fs_summary *)(kaddr + written_size); > @@ -4626,14 +4628,14 @@ static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr) > SUM_FOOTER_SIZE) > continue; > > - folio_mark_dirty(folio); > - f2fs_folio_put(folio, true); > - folio = NULL; > + f2fs_mark_cache_dirty(entry); > + f2fs_put_cache(entry, true); > + entry = NULL; > } > } > - if (folio) { > - folio_mark_dirty(folio); > - f2fs_folio_put(folio, true); > + if (entry) { > + f2fs_mark_cache_dirty(entry); > + f2fs_put_cache(entry, true); > } > } > > @@ -4687,29 +4689,29 @@ int f2fs_lookup_journal_in_cursum(struct f2fs_sb_info *sbi, > return -1; > } > > -static struct folio *get_current_sit_folio(struct f2fs_sb_info *sbi, > +static struct f2fs_cached_block *get_current_sit_cache(struct f2fs_sb_info *sbi, > unsigned int segno) > { > - return f2fs_get_meta_folio(sbi, current_sit_addr(sbi, segno)); > + return f2fs_get_meta_cache(sbi, current_sit_addr(sbi, segno)); > } > > -static struct folio *get_next_sit_folio(struct f2fs_sb_info *sbi, > +static struct f2fs_cached_block *get_next_sit_cache(struct f2fs_sb_info *sbi, > unsigned int start) > { > struct sit_info *sit_i = SIT_I(sbi); > - struct folio *folio; > + struct f2fs_cached_block *entry; > pgoff_t src_off, dst_off; > > src_off = current_sit_addr(sbi, start); > dst_off = next_sit_addr(sbi, src_off); > > - folio = f2fs_grab_meta_folio(sbi, dst_off); > - seg_info_to_sit_folio(sbi, folio, start); > + entry = f2fs_grab_meta_cache(sbi, dst_off); > + seg_info_to_sit_block(sbi, entry, start); > > - folio_mark_dirty(folio); > + f2fs_mark_cache_dirty(entry); > set_to_next_sit(sit_i, start); > > - return folio; > + return entry; > } > > static struct sit_entry_set *grab_sit_entry_set(void) > @@ -4839,8 +4841,9 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) > * #2, flush sit entries to sit page. > */ > list_for_each_entry_safe(ses, tmp, head, set_list) { > - struct folio *folio = NULL; > + struct f2fs_cached_block *entry = NULL; > struct f2fs_sit_block *raw_sit = NULL; > + > unsigned int start_segno = ses->start_segno; > unsigned int end = min(start_segno + SIT_ENTRY_PER_BLOCK, > (unsigned long)MAIN_SEGS(sbi)); > @@ -4854,8 +4857,8 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) > if (to_journal) { > down_write(&curseg->journal_rwsem); > } else { > - folio = get_next_sit_folio(sbi, start_segno); > - raw_sit = folio_address(folio); > + entry = get_next_sit_cache(sbi, start_segno); > + raw_sit = cache_address(entry); > } > > /* flush dirty sit entries in region of current sit set */ > @@ -4900,7 +4903,7 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) > if (to_journal) > up_write(&curseg->journal_rwsem); > else > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > > f2fs_bug_on(sbi, ses->entry_cnt); > release_sit_entry_set(ses); > @@ -5091,7 +5094,7 @@ static int build_sit_entries(struct f2fs_sb_info *sbi) > block_t sit_valid_blocks[2] = {0, 0}; > > do { > - readed = f2fs_ra_meta_pages(sbi, start_blk, BIO_MAX_VECS, > + readed = f2fs_ra_meta_caches(sbi, start_blk, BIO_MAX_VECS, > META_SIT, true); > > start = start_blk * sit_i->sents_per_block; > @@ -5099,15 +5102,15 @@ static int build_sit_entries(struct f2fs_sb_info *sbi) > > for (; start < end && start < MAIN_SEGS(sbi); start++) { > struct f2fs_sit_block *sit_blk; > - struct folio *folio; > + struct f2fs_cached_block *entry; > > se = &sit_i->sentries[start]; > - folio = get_current_sit_folio(sbi, start); > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > - sit_blk = folio_address(folio); > + entry = get_current_sit_cache(sbi, start); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > + sit_blk = cache_address(entry); > sit = sit_blk->entries[SIT_ENTRY_OFFSET(sit_i, start)]; > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > > err = check_block_count(sbi, start, &sit); > if (err) > diff --git a/fs/f2fs/segment.h b/fs/f2fs/segment.h > index db1079169a23..fff35c63a00f 100644 > --- a/fs/f2fs/segment.h > +++ b/fs/f2fs/segment.h > @@ -95,6 +95,8 @@ static inline void sanity_check_seg_type(struct f2fs_sb_info *sbi, > #define GET_SUM_BLKOFF(sbi, segno) (segno % (sbi)->sums_per_block) > #define SUM_BLK_PAGE_ADDR(sbi, folio, segno) \ > (folio_address(folio) + GET_SUM_BLKOFF(sbi, segno) * (sbi)->sum_blocksize) > +#define SUM_BLK_ENTRY_ADDR(sbi, entry, segno) \ > + (cache_address(entry) + GET_SUM_BLKOFF(sbi, segno) * (sbi)->sum_blocksize) > > #define GET_SUM_TYPE(footer) ((footer)->entry_type) > #define SET_SUM_TYPE(footer, type) ((footer)->entry_type = (type)) > @@ -417,8 +419,8 @@ static inline void __seg_info_to_raw_sit(struct seg_entry *se, > rs->mtime = cpu_to_le64(se->mtime); > } > > -static inline void seg_info_to_sit_folio(struct f2fs_sb_info *sbi, > - struct folio *folio, unsigned int start) > +static inline void seg_info_to_sit_block(struct f2fs_sb_info *sbi, > + struct f2fs_cached_block *entry, unsigned int start) > { > struct f2fs_sit_block *raw_sit; > struct seg_entry *se; > @@ -427,8 +429,8 @@ static inline void seg_info_to_sit_folio(struct f2fs_sb_info *sbi, > (unsigned long)MAIN_SEGS(sbi)); > int i; > > - raw_sit = folio_address(folio); > - memset(raw_sit, 0, PAGE_SIZE); > + raw_sit = cache_address(entry); > + memset(raw_sit, 0, sbi->blocksize); > for (i = 0; i < end - start; i++) { > rs = &raw_sit->entries[i]; > se = get_seg_entry(sbi, start + i); > @@ -991,6 +993,27 @@ static inline int nr_pages_to_skip(struct f2fs_sb_info *sbi, int type) > return 0; > } > > +/* > + * When writing cache asynchronously, align nr_to_write to BIO_MAX_VECS. > + */ > +static inline long adjust_flush_cache_number(struct f2fs_sb_info *sbi, int type) > +{ > + long nr_to_write; > + > + switch (type) { > + case META: > + nr_to_write = BIO_MAX_VECS; > + break; > + case NODE: > + nr_to_write = BIO_MAX_VECS << 1; > + break; > + default: > + f2fs_bug_on(sbi, 1); > + return 0; > + } > + return nr_to_write; > +} > + > /* > * When writing pages, it'd better align nr_to_write for segment size. > */ > diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c > index 89affe72f4fc..d3dee9b7d999 100644 > --- a/fs/f2fs/super.c > +++ b/fs/f2fs/super.c > @@ -1839,8 +1839,7 @@ static int f2fs_drop_inode(struct inode *inode) > * drop useless meta/node dirty pages. > */ > if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) { > - if (inode->i_ino == F2FS_NODE_INO(sbi) || > - inode->i_ino == F2FS_META_INO(sbi)) { > + if (inode->i_ino == F2FS_NODE_INO(sbi)) { > trace_f2fs_drop_inode(inode, 1); > return 1; > } > @@ -1942,8 +1941,7 @@ static void f2fs_dirty_inode(struct inode *inode, int flags) > { > struct f2fs_sb_info *sbi = F2FS_I_SB(inode); > > - if (inode->i_ino == F2FS_NODE_INO(sbi) || > - inode->i_ino == F2FS_META_INO(sbi)) > + if (inode->i_ino == F2FS_NODE_INO(sbi)) > return; > > if (is_inode_flag_set(inode, FI_AUTO_RECOVER)) > @@ -2039,9 +2037,10 @@ static void f2fs_put_super(struct super_block *sb) > > f2fs_wait_on_all_pages(sbi, F2FS_WB_CP_DATA); > > - if (err || f2fs_cp_error(sbi)) { > + if (err || f2fs_cp_error(sbi) || > + unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) { > truncate_inode_pages_final(NODE_MAPPING(sbi)); > - truncate_inode_pages_final(META_MAPPING(sbi)); > + f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); > } > > f2fs_bug_on(sbi, sbi->fsync_node_num); > @@ -2051,9 +2050,6 @@ static void f2fs_put_super(struct super_block *sb) > iput(sbi->node_inode); > sbi->node_inode = NULL; > > - iput(sbi->meta_inode); > - sbi->meta_inode = NULL; > - > f2fs_destroy_cache(META_CACHE(sbi)); > > /* Should check the page counts after dropping all node/meta pages */ > @@ -4383,7 +4379,6 @@ static void init_sb_info(struct f2fs_sb_info *sbi) > sbi->allocate_section_policy = ALLOCATE_FORWARD_NOHINT; > F2FS_ROOT_INO(sbi) = le32_to_cpu(raw_super->root_ino); > F2FS_NODE_INO(sbi) = le32_to_cpu(raw_super->node_ino); > - F2FS_META_INO(sbi) = le32_to_cpu(raw_super->meta_ino); > sbi->cur_victim_sec = NULL_SECNO; > sbi->gc_mode = GC_NORMAL; > sbi->next_victim_seg[BG_GC] = NULL_SEGNO; > @@ -5224,18 +5219,10 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) > if (err) > goto free_page_array_cache; > > - /* get an inode for meta space */ > - sbi->meta_inode = f2fs_iget(sb, F2FS_META_INO(sbi)); > - if (IS_ERR(sbi->meta_inode)) { > - f2fs_err(sbi, "Failed to read F2FS meta data inode"); > - err = PTR_ERR(sbi->meta_inode); > - goto free_meta_cache; > - } > - > err = f2fs_get_valid_checkpoint(sbi); > if (err) { > f2fs_err(sbi, "Failed to get valid F2FS checkpoint"); > - goto free_meta_inode; > + goto free_meta_cache; > } > > if (__is_set_ckpt_flags(F2FS_CKPT(sbi), CP_QUOTA_NEED_FSCK_FLAG)) > @@ -5529,7 +5516,7 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) > * followed by f2fs_write_checkpoint() through f2fs_write_node_pages(), which > * falls into an infinite loop in f2fs_sync_meta_pages(). > */ > - truncate_inode_pages_final(META_MAPPING(sbi)); > + f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); > /* evict some inodes being cached by GC */ > evict_inodes(sb); > f2fs_unregister_sysfs(sbi); > @@ -5559,10 +5546,6 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) > free_devices: > destroy_device_list(sbi); > kvfree(sbi->ckpt); > -free_meta_inode: > - make_bad_inode(sbi->meta_inode); > - iput(sbi->meta_inode); > - sbi->meta_inode = NULL; > free_meta_cache: > f2fs_destroy_cache(META_CACHE(sbi)); > free_page_array_cache: > diff --git a/include/linux/f2fs_fs.h b/include/linux/f2fs_fs.h > index bb2b6cd5d507..0c027d00a1ea 100644 > --- a/include/linux/f2fs_fs.h > +++ b/include/linux/f2fs_fs.h > @@ -35,7 +35,6 @@ > > #define F2FS_ROOT_INO(sbi) ((sbi)->root_ino_num) > #define F2FS_NODE_INO(sbi) ((sbi)->node_ino_num) > -#define F2FS_META_INO(sbi) ((sbi)->meta_ino_num) > #define F2FS_COMPRESS_INO(sbi) (NM_I(sbi)->max_nid) > > #define F2FS_MAX_QUOTAS 3 > -- > 2.49.0 > > > > _______________________________________________ > Linux-f2fs-devel mailing list > Linux-f2fs-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [f2fs-dev] [PATCH v1 05/12] f2fs: cache: use meta cache @ 2026-08-20 5:11 ` Jaegeuk Kim 0 siblings, 0 replies; 34+ messages in thread From: Jaegeuk Kim @ 2026-08-20 5:11 UTC (permalink / raw) To: Chao Yu; +Cc: linux-kernel, linux-f2fs-devel On 08/20, Chao Yu via Linux-f2fs-devel wrote: > This patch migrates F2FS meta block caching from the fake VFS inode > page cache (sbi->meta_inode) to meta cache (sbi->meta_blocks). > > It converts CP, SIT, NAT, SSA, recovery, and GC metadata I/O paths to > operate on struct f2fs_cached_block instead of folio, and removes > sbi->meta_inode. > > Signed-off-by: Chao Yu <chao@kernel.org> > --- > fs/f2fs/cache.c | 2 + > fs/f2fs/checkpoint.c | 394 ++++++++++++++++++---------------------- > fs/f2fs/compress.c | 4 +- > fs/f2fs/data.c | 29 ++- > fs/f2fs/debug.c | 12 +- > fs/f2fs/f2fs.h | 81 +++------ > fs/f2fs/file.c | 4 +- > fs/f2fs/gc.c | 134 +++++++------- > fs/f2fs/inline.c | 9 +- > fs/f2fs/inode.c | 9 +- > fs/f2fs/node.c | 125 +++++++------ > fs/f2fs/node.h | 4 +- > fs/f2fs/recovery.c | 198 ++++++++++---------- > fs/f2fs/segment.c | 201 ++++++++++---------- > fs/f2fs/segment.h | 31 +++- > fs/f2fs/super.c | 31 +--- > include/linux/f2fs_fs.h | 1 - > 17 files changed, 606 insertions(+), 663 deletions(-) > > diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c > index cb5b26046162..afef808e485a 100644 > --- a/fs/f2fs/cache.c > +++ b/fs/f2fs/cache.c > @@ -631,6 +631,8 @@ static int f2fs_cache_writeback_kthread(void *data) > break; > if (f2fs_cp_error(sbi)) > continue; > + > + f2fs_write_meta_caches(sbi); > } > return 0; > } > diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c > index 729d19680caf..1a7083540b82 100644 > --- a/fs/f2fs/checkpoint.c > +++ b/fs/f2fs/checkpoint.c > @@ -17,6 +17,7 @@ > #include <linux/delayacct.h> > #include <linux/ioprio.h> > #include <linux/math64.h> > +#include <linux/freezer.h> > > #include "f2fs.h" > #include "node.h" > @@ -235,27 +236,27 @@ struct kmem_cache *f2fs_inode_entry_slab; > /* > * We guarantee no failure on the returned page. > */ > -struct folio *f2fs_grab_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index) > +struct f2fs_cached_block *f2fs_grab_meta_cache(struct f2fs_sb_info *sbi, > + pgoff_t index) > { > - struct address_space *mapping = META_MAPPING(sbi); > - struct folio *folio; > + struct f2fs_cached_block *entry; > repeat: > - folio = f2fs_grab_cache_folio(mapping, index, false); > - if (IS_ERR(folio)) { > + entry = f2fs_grab_cache(META_CACHE(sbi), index, > + F2FS_CACHE_LOCK_CREATE); > + if (IS_ERR(entry)) { > cond_resched(); > goto repeat; > } > - f2fs_folio_wait_writeback(folio, META, true, true); > - if (!folio_test_uptodate(folio)) > - folio_mark_uptodate(folio); > - return folio; > + f2fs_cache_wait_writeback(entry); > + if (!f2fs_cache_test_uptodate(entry)) > + f2fs_cache_set_uptodate(entry); > + return entry; > } > > -static struct folio *__get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index, > - bool is_meta) > +static struct f2fs_cached_block *__get_meta_cache(struct f2fs_sb_info *sbi, > + pgoff_t index, bool is_meta) > { > - struct address_space *mapping = META_MAPPING(sbi); > - struct folio *folio; > + struct f2fs_cached_block *entry; > struct f2fs_io_info fio = { > .sbi = sbi, > .type = META, > @@ -265,70 +266,75 @@ static struct folio *__get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index, > .new_blkaddr = index, > .encrypted_page = NULL, > .is_por = !is_meta ? 1 : 0, > + .is_cache = 1, > }; > int err; > > if (unlikely(!is_meta)) > fio.op_flags &= ~REQ_META; > repeat: > - folio = f2fs_grab_cache_folio(mapping, index, false); > - if (IS_ERR(folio)) { > + entry = f2fs_grab_cache(META_CACHE(sbi), index, > + F2FS_CACHE_LOCK_CREATE); > + if (IS_ERR(entry)) { > cond_resched(); > goto repeat; > } > - if (folio_test_uptodate(folio)) > + if (f2fs_cache_test_uptodate(entry)) > goto out; > > - fio.folio = folio; > + fio.cache_entry = entry; > > - err = f2fs_submit_page_bio(&fio); > + err = f2fs_submit_cache_read(&fio); > if (err) { > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > return ERR_PTR(err); > } > > f2fs_update_iostat(sbi, NULL, FS_META_READ_IO, F2FS_BLKSIZE); > > - folio_lock(folio); > - if (unlikely(!is_meta_folio(folio))) { > - f2fs_folio_put(folio, true); > + f2fs_lock_cache(entry); > + if (unlikely(!f2fs_is_meta_cache(entry))) { > + f2fs_put_cache(entry, true); > goto repeat; > } > > - if (unlikely(!folio_test_uptodate(folio))) { > - f2fs_handle_page_eio(sbi, folio, META); > - f2fs_folio_put(folio, true); > + if (unlikely(!f2fs_cache_test_uptodate(entry))) { > + f2fs_handle_page_eio(sbi, entry->index, META); > + f2fs_put_cache(entry, true); > return ERR_PTR(-EIO); > } > out: > - return folio; > + return entry; > } > > -struct folio *f2fs_get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index) > +struct f2fs_cached_block *f2fs_get_meta_cache(struct f2fs_sb_info *sbi, > + pgoff_t index) > { > - return __get_meta_folio(sbi, index, true); > + return __get_meta_cache(sbi, index, true); > } > > -struct folio *f2fs_get_meta_folio_retry(struct f2fs_sb_info *sbi, pgoff_t index) > +struct f2fs_cached_block *f2fs_get_meta_cache_retry(struct f2fs_sb_info *sbi, > + pgoff_t index) > { > - struct folio *folio; > + struct f2fs_cached_block *entry; > int count = 0; > > retry: > - folio = __get_meta_folio(sbi, index, true); > - if (IS_ERR(folio)) { > - if (PTR_ERR(folio) == -EIO && > + entry = __get_meta_cache(sbi, index, true); > + if (IS_ERR(entry)) { > + if (PTR_ERR(entry) == -EIO && > ++count <= DEFAULT_RETRY_IO_COUNT) > goto retry; > f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_META_PAGE); > } > - return folio; > + return entry; > } > > /* for POR only */ > -struct folio *f2fs_get_tmp_folio(struct f2fs_sb_info *sbi, pgoff_t index) > +struct f2fs_cached_block *f2fs_get_tmp_cache(struct f2fs_sb_info *sbi, > + pgoff_t index) > { > - return __get_meta_folio(sbi, index, false); > + return __get_meta_cache(sbi, index, false); > } > > static bool __is_bitmap_valid(struct f2fs_sb_info *sbi, block_t blkaddr, > @@ -446,9 +452,10 @@ bool f2fs_is_valid_blkaddr_raw(struct f2fs_sb_info *sbi, > /* > * Readahead CP/NAT/SIT/SSA/POR pages > */ > -int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, > - int type, bool sync) > +int f2fs_ra_meta_caches(struct f2fs_sb_info *sbi, block_t start, > + int nrpages, int type, bool sync) > { > + struct f2fs_cached_block_list *cache = META_CACHE(sbi); > block_t blkno = start; > struct f2fs_io_info fio = { > .sbi = sbi, > @@ -458,6 +465,7 @@ int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, > .encrypted_page = NULL, > .in_list = 0, > .is_por = (type == META_POR) ? 1 : 0, > + .is_cache = 1, > }; > struct blk_plug plug; > int err; > @@ -467,7 +475,7 @@ int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, > > blk_start_plug(&plug); > for (; nrpages-- > 0; blkno++) { > - struct folio *folio; > + struct f2fs_cached_block *entry; > > if (!f2fs_is_valid_blkaddr(sbi, blkno, type)) > goto out; > @@ -494,62 +502,58 @@ int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, > fio.new_blkaddr = blkno; > break; > default: > - BUG(); > + f2fs_bug_on(sbi, 1); > } > > - folio = f2fs_grab_cache_folio(META_MAPPING(sbi), > - fio.new_blkaddr, false); > - if (IS_ERR(folio)) > + entry = f2fs_grab_cache(cache, fio.new_blkaddr, > + F2FS_CACHE_LOCK_CREATE); > + if (IS_ERR(entry)) > continue; > - if (folio_test_uptodate(folio)) { > - f2fs_folio_put(folio, true); > + if (f2fs_cache_test_uptodate(entry)) { > + f2fs_put_cache(entry, true); > continue; > } > > - fio.folio = folio; > - err = f2fs_submit_page_bio(&fio); > - f2fs_folio_put(folio, err ? true : false); > + fio.cache_entry = entry; > + err = f2fs_submit_cache_read(&fio); > + f2fs_put_cache(entry, err ? true : false); > > if (!err) > f2fs_update_iostat(sbi, NULL, FS_META_READ_IO, > - F2FS_BLKSIZE); > + sbi->blocksize); > } > out: > blk_finish_plug(&plug); > return blkno - start; > } > > -void f2fs_ra_meta_pages_cond(struct f2fs_sb_info *sbi, pgoff_t index, > - unsigned int ra_blocks) > +void f2fs_ra_meta_caches_cond(struct f2fs_sb_info *sbi, pgoff_t index, > + unsigned int ra_blocks) > { > - struct folio *folio; > + struct f2fs_cached_block *entry; > bool readahead = false; > > if (ra_blocks == RECOVERY_MIN_RA_BLOCKS) > return; > > - folio = filemap_get_folio(META_MAPPING(sbi), index); > - if (IS_ERR(folio) || !folio_test_uptodate(folio)) > + entry = f2fs_find_cache(META_CACHE(sbi), index); > + if (IS_ERR(entry) || !f2fs_cache_test_uptodate(entry)) > readahead = true; > - f2fs_folio_put(folio, false); > + f2fs_put_cache(entry, false); > > if (readahead) > - f2fs_ra_meta_pages(sbi, index, ra_blocks, META_POR, true); > + f2fs_ra_meta_caches(sbi, index, ra_blocks, META_POR, true); > } > > -static bool __f2fs_write_meta_folio(struct folio *folio, > - struct writeback_control *wbc, > +static bool __f2fs_write_meta_cache(struct f2fs_cached_block *entry, > enum iostat_type io_type) > { > - struct f2fs_sb_info *sbi = F2FS_F_SB(folio); > - > - trace_f2fs_writepage(folio, META); > + struct f2fs_sb_info *sbi = entry->cache->sbi; > > if (unlikely(f2fs_cp_error(sbi))) { > if (is_sbi_flag_set(sbi, SBI_IS_CLOSE)) { > - folio_clear_uptodate(folio); > - dec_page_count(sbi, F2FS_DIRTY_META); > - folio_unlock(folio); > + f2fs_force_clear_cache_dirty(entry); > + f2fs_unlock_cache(entry); > return true; > } > goto redirty_out; > @@ -557,10 +561,10 @@ static bool __f2fs_write_meta_folio(struct folio *folio, > if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING))) > goto redirty_out; > > - f2fs_do_write_meta_page(sbi, folio, io_type); > + f2fs_do_write_meta_cache(sbi, entry, io_type); > dec_page_count(sbi, F2FS_DIRTY_META); > > - folio_unlock(folio); > + f2fs_unlock_cache(entry); > > if (unlikely(f2fs_cp_error(sbi))) > f2fs_submit_merged_write(sbi, META); > @@ -568,101 +572,89 @@ static bool __f2fs_write_meta_folio(struct folio *folio, > return true; > > redirty_out: > - folio_redirty_for_writepage(wbc, folio); > + f2fs_cache_set_dirty(entry); > return false; > } > > -static int f2fs_write_meta_pages(struct address_space *mapping, > - struct writeback_control *wbc) > +void f2fs_write_meta_caches(struct f2fs_sb_info *sbi) > { > - struct f2fs_sb_info *sbi = F2FS_M_SB(mapping); > struct f2fs_lock_context lc; > - long diff, written; > + long nr_to_write = LONG_MAX; > > if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING))) > - goto skip_write; > + return; > > - /* collect a number of dirty meta pages and write together */ > - if (wbc->sync_mode != WB_SYNC_ALL && > - get_pages(sbi, F2FS_DIRTY_META) < > - nr_pages_to_skip(sbi, META)) > - goto skip_write; > + /* collect a number of dirty meta caches and write together */ > + if (get_pages(sbi, F2FS_DIRTY_META) < > + nr_pages_to_skip(sbi, META)) > + return; > > - /* if locked failed, cp will flush dirty pages instead */ > + /* if locked failed, cp will flush dirty caches instead */ > if (!f2fs_down_write_trylock_trace(&sbi->cp_global_sem, &lc)) > - goto skip_write; > + return; > > - trace_f2fs_writepages(mapping->host, wbc, META); > - diff = nr_pages_to_write(sbi, META, wbc); > - written = f2fs_sync_meta_pages(sbi, wbc->nr_to_write, FS_META_IO); > + nr_to_write = adjust_flush_cache_number(sbi, META); > + f2fs_sync_meta_caches(sbi, nr_to_write, false, FS_META_IO); > f2fs_up_write_trace(&sbi->cp_global_sem, &lc); > - wbc->nr_to_write = max((long)0, wbc->nr_to_write - written - diff); > - return 0; > - > -skip_write: > - wbc->pages_skipped += get_pages(sbi, F2FS_DIRTY_META); > - trace_f2fs_writepages(mapping->host, wbc, META); > - return 0; > } > > -long f2fs_sync_meta_pages(struct f2fs_sb_info *sbi, long nr_to_write, > - enum iostat_type io_type) > +long f2fs_sync_meta_caches(struct f2fs_sb_info *sbi, long nr_to_write, > + bool sync, enum iostat_type io_type) > { > - struct address_space *mapping = META_MAPPING(sbi); > pgoff_t index = 0, prev = ULONG_MAX; > - struct folio_batch fbatch; > + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; > long nwritten = 0; > - int nr_folios; > - struct writeback_control wbc = {}; > + int nr; > struct blk_plug plug; > - > - folio_batch_init(&fbatch); > + bool background = nr_to_write != LONG_MAX; > > blk_start_plug(&plug); > > - while ((nr_folios = filemap_get_folios_tag(mapping, &index, > - (pgoff_t)-1, > - PAGECACHE_TAG_DIRTY, &fbatch))) { > + while ((nr = f2fs_cache_gang_lookup_tag(META_CACHE(sbi), entries, > + &index, F2FS_ONSTACK_CACHES, F2FS_CACHE_TAG_DIRTY))) { > int i; > > - for (i = 0; i < nr_folios; i++) { > - struct folio *folio = fbatch.folios[i]; > + for (i = 0; i < nr; i++) { > + struct f2fs_cached_block *entry = entries[i]; > + > + if (background && unlikely(freezing(current))) { > + f2fs_cache_gang_release(entries, nr); > + goto stop; > + } > > - if (nr_to_write != LONG_MAX && i != 0 && > - folio->index != prev + > - folio_nr_pages(fbatch.folios[i-1])) { > - folio_batch_release(&fbatch); > + if (background && i != 0 && > + entry->index != prev + 1) { > + f2fs_cache_gang_release(entries, nr); > goto stop; > } > > - folio_lock(folio); > + f2fs_lock_cache(entry); > > - if (unlikely(!is_meta_folio(folio))) { > + if (unlikely(!f2fs_is_meta_cache(entry))) { > continue_unlock: > - folio_unlock(folio); > + f2fs_unlock_cache(entry); > continue; > } > - if (!folio_test_dirty(folio)) { > + if (!f2fs_cache_test_dirty(entry)) { > /* someone wrote it for us */ > goto continue_unlock; > } > > - f2fs_folio_wait_writeback(folio, META, true, true); > + f2fs_cache_wait_writeback(entry); > > - if (!folio_clear_dirty_for_io(folio)) > + if (!f2fs_clear_cache_dirty(entry)) > goto continue_unlock; > > - if (!__f2fs_write_meta_folio(folio, &wbc, > - io_type)) { > - folio_unlock(folio); > + if (!__f2fs_write_meta_cache(entry, io_type)) { > + f2fs_unlock_cache(entry); > break; > } > - nwritten += folio_nr_pages(folio); > - prev = folio->index; > + nwritten += sbi->blocksize; > + prev = entry->index; > if (unlikely(nwritten >= nr_to_write)) > break; > } > - folio_batch_release(&fbatch); > + f2fs_cache_gang_release(entries, nr); > cond_resched(); > } > stop: > @@ -674,29 +666,6 @@ long f2fs_sync_meta_pages(struct f2fs_sb_info *sbi, long nr_to_write, > return nwritten; > } > > -static bool f2fs_dirty_meta_folio(struct address_space *mapping, > - struct folio *folio) > -{ > - trace_f2fs_set_page_dirty(folio, META); > - > - if (!folio_test_uptodate(folio)) > - folio_mark_uptodate(folio); > - if (filemap_dirty_folio(mapping, folio)) { > - inc_page_count(F2FS_M_SB(mapping), F2FS_DIRTY_META); > - folio_set_f2fs_reference(folio); > - return true; > - } > - return false; > -} > - > -const struct address_space_operations f2fs_meta_aops = { > - .writepages = f2fs_write_meta_pages, > - .dirty_folio = f2fs_dirty_meta_folio, > - .invalidate_folio = f2fs_invalidate_folio, > - .release_folio = f2fs_release_folio, > - .migrate_folio = filemap_migrate_folio, > -}; > - > static void __add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, > unsigned int devidx, int type) > { > @@ -1036,20 +1005,20 @@ int f2fs_recover_orphan_inodes(struct f2fs_sb_info *sbi) > start_blk = __start_cp_addr(sbi) + 1 + __cp_payload(sbi); > orphan_blocks = __start_sum_addr(sbi) - 1 - __cp_payload(sbi); > > - f2fs_ra_meta_pages(sbi, start_blk, orphan_blocks, META_CP, true); > + f2fs_ra_meta_caches(sbi, start_blk, orphan_blocks, META_CP, true); > > for (i = 0; i < orphan_blocks; i++) { > - struct folio *folio; > + struct f2fs_cached_block *entry; > struct f2fs_orphan_block *orphan_blk; > unsigned int entry_count; > > - folio = f2fs_get_meta_folio(sbi, start_blk + i); > - if (IS_ERR(folio)) { > - err = PTR_ERR(folio); > + entry = f2fs_get_meta_cache(sbi, start_blk + i); > + if (IS_ERR(entry)) { > + err = PTR_ERR(entry); > goto out; > } > > - orphan_blk = folio_address(folio); > + orphan_blk = cache_address(entry); > entry_count = le32_to_cpu(orphan_blk->entry_count); > if (entry_count > F2FS_ORPHANS_PER_BLOCK) { > f2fs_err(sbi, "invalid orphan inode entry count %u", > @@ -1057,7 +1026,7 @@ int f2fs_recover_orphan_inodes(struct f2fs_sb_info *sbi) > set_sbi_flag(sbi, SBI_NEED_FSCK); > f2fs_handle_error(sbi, ERROR_INCONSISTENT_ORPHAN); > err = -EFSCORRUPTED; > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > goto out; > } > > @@ -1066,11 +1035,11 @@ int f2fs_recover_orphan_inodes(struct f2fs_sb_info *sbi) > > err = recover_orphan_inode(sbi, ino); > if (err) { > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > goto out; > } > } > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > } > /* clear Orphan Flag */ > clear_ckpt_flags(sbi, CP_ORPHAN_PRESENT_FLAG); > @@ -1087,9 +1056,9 @@ static void write_orphan_inodes(struct f2fs_sb_info *sbi, block_t start_blk) > unsigned int nentries = 0; > unsigned short index = 1; > unsigned short orphan_blocks; > - struct folio *folio = NULL; > struct ino_entry *orphan = NULL; > struct inode_management *im = &sbi->im[ORPHAN_INO]; > + struct f2fs_cached_block *entry = NULL; > > orphan_blocks = GET_ORPHAN_BLOCKS(im->ino_num); > > @@ -1102,9 +1071,9 @@ static void write_orphan_inodes(struct f2fs_sb_info *sbi, block_t start_blk) > > /* loop for each orphan inode entry and write them in journal block */ > list_for_each_entry(orphan, head, list) { > - if (!folio) { > - folio = f2fs_grab_meta_folio(sbi, start_blk++); > - orphan_blk = folio_address(folio); > + if (!entry) { > + entry = f2fs_grab_meta_cache(sbi, start_blk++); > + orphan_blk = (struct f2fs_orphan_block *)cache_address(entry); > memset(orphan_blk, 0, sizeof(*orphan_blk)); > } > > @@ -1119,20 +1088,20 @@ static void write_orphan_inodes(struct f2fs_sb_info *sbi, block_t start_blk) > orphan_blk->blk_addr = cpu_to_le16(index); > orphan_blk->blk_count = cpu_to_le16(orphan_blocks); > orphan_blk->entry_count = cpu_to_le32(nentries); > - folio_mark_dirty(folio); > - f2fs_folio_put(folio, true); > + f2fs_mark_cache_dirty(entry); > + f2fs_put_cache(entry, true); > index++; > nentries = 0; > - folio = NULL; > + entry = NULL; > } > } > > - if (folio) { > + if (entry) { > orphan_blk->blk_addr = cpu_to_le16(index); > orphan_blk->blk_count = cpu_to_le16(orphan_blocks); > orphan_blk->entry_count = cpu_to_le32(nentries); > - folio_mark_dirty(folio); > - f2fs_folio_put(folio, true); > + f2fs_mark_cache_dirty(entry); > + f2fs_put_cache(entry, true); > } > } > > @@ -1151,29 +1120,29 @@ static __u32 f2fs_checkpoint_chksum(struct f2fs_checkpoint *ckpt) > } > > static int get_checkpoint_version(struct f2fs_sb_info *sbi, block_t cp_addr, > - struct f2fs_checkpoint **cp_block, struct folio **cp_folio, > + struct f2fs_checkpoint **cp_block, struct f2fs_cached_block **cp_entry, > unsigned long long *version) > { > size_t crc_offset = 0; > __u32 crc; > > - *cp_folio = f2fs_get_meta_folio(sbi, cp_addr); > - if (IS_ERR(*cp_folio)) > - return PTR_ERR(*cp_folio); > + *cp_entry = f2fs_get_meta_cache(sbi, cp_addr); > + if (IS_ERR(*cp_entry)) > + return PTR_ERR(*cp_entry); > > - *cp_block = folio_address(*cp_folio); > + *cp_block = (struct f2fs_checkpoint *)cache_address(*cp_entry); > > crc_offset = le32_to_cpu((*cp_block)->checksum_offset); > if (crc_offset < CP_MIN_CHKSUM_OFFSET || > crc_offset > CP_CHKSUM_OFFSET) { > - f2fs_folio_put(*cp_folio, true); > + f2fs_put_cache(*cp_entry, true); > f2fs_warn(sbi, "invalid crc_offset: %zu", crc_offset); > return -EINVAL; > } > > crc = f2fs_checkpoint_chksum(*cp_block); > if (crc != cur_cp_crc(*cp_block)) { > - f2fs_folio_put(*cp_folio, true); > + f2fs_put_cache(*cp_entry, true); > f2fs_warn(sbi, "invalid crc value"); > return -EINVAL; > } > @@ -1182,17 +1151,17 @@ static int get_checkpoint_version(struct f2fs_sb_info *sbi, block_t cp_addr, > return 0; > } > > -static struct folio *validate_checkpoint(struct f2fs_sb_info *sbi, > +static struct f2fs_cached_block *validate_checkpoint(struct f2fs_sb_info *sbi, > block_t cp_addr, unsigned long long *version) > { > - struct folio *cp_folio_1 = NULL, *cp_folio_2 = NULL; > + struct f2fs_cached_block *cp_entry_1 = NULL, *cp_entry_2 = NULL; > struct f2fs_checkpoint *cp_block = NULL; > unsigned long long cur_version = 0, pre_version = 0; > unsigned int cp_blocks; > int err; > > err = get_checkpoint_version(sbi, cp_addr, &cp_block, > - &cp_folio_1, version); > + &cp_entry_1, version); > if (err) > return NULL; > > @@ -1207,19 +1176,20 @@ static struct folio *validate_checkpoint(struct f2fs_sb_info *sbi, > > cp_addr += cp_blocks - 1; > err = get_checkpoint_version(sbi, cp_addr, &cp_block, > - &cp_folio_2, version); > + &cp_entry_2, version); > if (err) > goto invalid_cp; > cur_version = *version; > > if (cur_version == pre_version) { > *version = cur_version; > - f2fs_folio_put(cp_folio_2, true); > - return cp_folio_1; > + f2fs_put_cache(cp_entry_2, true); > + return cp_entry_1; > } > - f2fs_folio_put(cp_folio_2, true); > + f2fs_put_cache(cp_entry_2, true); > invalid_cp: > - f2fs_folio_put(cp_folio_1, true); > + if (cp_entry_1) > + f2fs_put_cache(cp_entry_1, true); > return NULL; > } > > @@ -1227,7 +1197,7 @@ int f2fs_get_valid_checkpoint(struct f2fs_sb_info *sbi) > { > struct f2fs_checkpoint *cp_block; > struct f2fs_super_block *fsb = sbi->raw_super; > - struct folio *cp1, *cp2, *cur_folio; > + struct f2fs_cached_block *cp1 = NULL, *cp2 = NULL, *cur_entry; > unsigned long blk_size = sbi->blocksize; > unsigned long long cp1_version = 0, cp2_version = 0; > unsigned long long cp_start_blk_no; > @@ -1254,22 +1224,22 @@ int f2fs_get_valid_checkpoint(struct f2fs_sb_info *sbi) > > if (cp1 && cp2) { > if (ver_after(cp2_version, cp1_version)) > - cur_folio = cp2; > + cur_entry = cp2; > else > - cur_folio = cp1; > + cur_entry = cp1; > } else if (cp1) { > - cur_folio = cp1; > + cur_entry = cp1; > } else if (cp2) { > - cur_folio = cp2; > + cur_entry = cp2; > } else { > err = -EFSCORRUPTED; > goto fail_no_cp; > } > > - cp_block = folio_address(cur_folio); > + cp_block = (struct f2fs_checkpoint *)cache_address(cur_entry); > memcpy(sbi->ckpt, cp_block, blk_size); > > - if (cur_folio == cp1) > + if (cur_entry == cp1) > sbi->cur_cp_pack = 1; > else > sbi->cur_cp_pack = 2; > @@ -1284,30 +1254,35 @@ int f2fs_get_valid_checkpoint(struct f2fs_sb_info *sbi) > goto done; > > cp_blk_no = le32_to_cpu(fsb->cp_blkaddr); > - if (cur_folio == cp2) > + if (cur_entry == cp2) > cp_blk_no += BIT(le32_to_cpu(fsb->log_blocks_per_seg)); > > for (i = 1; i < cp_blks; i++) { > + struct f2fs_cached_block *cur_entry_payload; > void *sit_bitmap_ptr; > unsigned char *ckpt = (unsigned char *)sbi->ckpt; > > - cur_folio = f2fs_get_meta_folio(sbi, cp_blk_no + i); > - if (IS_ERR(cur_folio)) { > - err = PTR_ERR(cur_folio); > + cur_entry_payload = f2fs_get_meta_cache(sbi, cp_blk_no + i); > + if (IS_ERR(cur_entry_payload)) { > + err = PTR_ERR(cur_entry_payload); > goto free_fail_no_cp; > } > - sit_bitmap_ptr = folio_address(cur_folio); > + sit_bitmap_ptr = cache_address(cur_entry_payload); > memcpy(ckpt + i * blk_size, sit_bitmap_ptr, blk_size); > - f2fs_folio_put(cur_folio, true); > + f2fs_put_cache(cur_entry_payload, true); > } > done: > - f2fs_folio_put(cp1, true); > - f2fs_folio_put(cp2, true); > + if (cp1) > + f2fs_put_cache(cp1, true); In f2fs_put_cache(x, y), if (!x) return; > + if (cp2) > + f2fs_put_cache(cp2, true); > return 0; > > free_fail_no_cp: > - f2fs_folio_put(cp1, true); > - f2fs_folio_put(cp2, true); > + if (cp1) > + f2fs_put_cache(cp1, true); > + if (cp2) > + f2fs_put_cache(cp2, true); > fail_no_cp: > kvfree(sbi->ckpt); > return err; > @@ -1621,7 +1596,7 @@ void f2fs_wait_on_all_pages(struct f2fs_sb_info *sbi, int type) > break; > > if (type == F2FS_DIRTY_META) > - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_CP_META_IO); > + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_META_IO); > else if (type == F2FS_WB_CP_DATA) > f2fs_submit_merged_write(sbi, DATA); > > @@ -1700,31 +1675,24 @@ static void update_ckpt_flags(struct f2fs_sb_info *sbi, struct cp_control *cpc) > static void commit_checkpoint(struct f2fs_sb_info *sbi, > void *src, block_t blk_addr) > { > - struct writeback_control wbc = {}; > - > - /* > - * filemap_get_folios_tag and folio_lock again will take > - * some extra time. Therefore, f2fs_update_meta_pages and > - * f2fs_sync_meta_pages are combined in this function. > - */ > - struct folio *folio = f2fs_grab_meta_folio(sbi, blk_addr); > + struct f2fs_cached_block *entry = f2fs_grab_meta_cache(sbi, blk_addr); > > - memcpy(folio_address(folio), src, PAGE_SIZE); > + memcpy(cache_address(entry), src, F2FS_BLKSIZE); > > - folio_mark_dirty(folio); > - if (unlikely(!folio_clear_dirty_for_io(folio))) > + f2fs_mark_cache_dirty(entry); > + if (unlikely(!f2fs_clear_cache_dirty(entry))) > f2fs_bug_on(sbi, 1); > > /* writeout cp pack 2 page */ > - if (unlikely(!__f2fs_write_meta_folio(folio, &wbc, FS_CP_META_IO))) { > + if (unlikely(!__f2fs_write_meta_cache(entry, FS_CP_META_IO))) { > if (f2fs_cp_error(sbi)) { > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > return; > } > f2fs_bug_on(sbi, true); > } > > - f2fs_folio_put(folio, false); > + f2fs_put_cache(entry, false); > > /* submit checkpoint (with barrier if NOBARRIER is not set) */ > f2fs_submit_merged_write(sbi, META_FLUSH); > @@ -1793,7 +1761,7 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc) > int err; > > /* Flush all the NAT/SIT pages */ > - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_CP_META_IO); > + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_CP_META_IO); > > stat_cp_time(cpc, CP_TIME_SYNC_META); > > @@ -1892,7 +1860,7 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc) > } > > /* Here, we have one bio having CP pack except cp pack 2 page */ > - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_CP_META_IO); > + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_CP_META_IO); > stat_cp_time(cpc, CP_TIME_SYNC_CP_META); > > /* Wait for all dirty meta pages to be submitted for IO */ > @@ -1919,10 +1887,10 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc) > * used for migration of encrypted, verity or compressed inode's blocks. > */ > if (f2fs_sb_has_encrypt(sbi) || f2fs_sb_has_verity(sbi) || > - f2fs_sb_has_compression(sbi)) > - f2fs_bug_on(sbi, > - invalidate_inode_pages2_range(META_MAPPING(sbi), > - MAIN_BLKADDR(sbi), MAX_BLKADDR(sbi) - 1)); > + f2fs_sb_has_compression(sbi)) { > + f2fs_truncate_meta_caches(sbi, MAIN_BLKADDR(sbi), > + MAX_BLKADDR(sbi) - MAIN_BLKADDR(sbi)); > + } > > f2fs_release_ino_entry(sbi, false); > > diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c > index 91855d91bbdd..bb749f6257a1 100644 > --- a/fs/f2fs/compress.c > +++ b/fs/f2fs/compress.c > @@ -1150,7 +1150,7 @@ static int prepare_compress_overwrite(struct compress_ctx *cc, > f2fs_compress_ctx_add_page(cc, folio); > > if (!folio_test_uptodate(folio)) { > - f2fs_handle_page_eio(sbi, folio, DATA); > + f2fs_handle_page_eio(sbi, folio->index, DATA); > release_and_retry: > f2fs_put_rpages(cc); > f2fs_unlock_rpages(cc, i + 1); > @@ -1359,7 +1359,7 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc, > fio.old_blkaddr = data_blkaddr(dn.inode, dn.node_folio, > dn.ofs_in_node + i + 1); > > - /* wait for GCed page writeback via META_MAPPING */ > + /* wait for GCed page writeback via generic cache */ > f2fs_wait_on_block_writeback(inode, fio.old_blkaddr); > > if (fio.encrypted) { > diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c > index 110282bb8dcd..92c3293f0a1e 100644 > --- a/fs/f2fs/data.c > +++ b/fs/f2fs/data.c > @@ -65,8 +65,7 @@ bool f2fs_is_cp_guaranteed(const struct folio *folio) > > inode = mapping->host; > > - if (inode->i_ino == F2FS_META_INO(sbi) || > - inode->i_ino == F2FS_NODE_INO(sbi) || > + if (inode->i_ino == F2FS_NODE_INO(sbi) || > S_ISDIR(inode->i_mode)) > return true; > > @@ -84,9 +83,6 @@ static enum count_type __read_io_type(struct folio *folio) > struct inode *inode = mapping->host; > struct f2fs_sb_info *sbi = F2FS_I_SB(inode); > > - if (inode->i_ino == F2FS_META_INO(sbi)) > - return F2FS_RD_META; > - > if (inode->i_ino == F2FS_NODE_INO(sbi)) > return F2FS_RD_NODE; > } > @@ -1452,7 +1448,7 @@ static void f2fs_submit_page_read(struct inode *inode, struct fsverity_info *vi, > bio = f2fs_grab_read_bio(inode, vi, blkaddr, 1, op_flags, folio->index, > for_write); > > - /* wait for GCed page writeback via META_MAPPING */ > + /* wait for GCed page writeback via generic cache */ > f2fs_wait_on_block_writeback(inode, blkaddr); > > if (!bio_add_folio(bio, folio, PAGE_SIZE, 0)) > @@ -3110,7 +3106,7 @@ static void f2fs_readahead(struct readahead_control *rac) > int f2fs_encrypt_one_page(struct f2fs_io_info *fio) > { > struct inode *inode = fio_inode(fio); > - struct folio *mfolio; > + struct f2fs_cached_block *entry; > struct page *page; > > if (!f2fs_encrypted_file(inode)) > @@ -3126,12 +3122,13 @@ int f2fs_encrypt_one_page(struct f2fs_io_info *fio) > if (IS_ERR(fio->encrypted_page)) > return PTR_ERR(fio->encrypted_page); > > - mfolio = filemap_lock_folio(META_MAPPING(fio->sbi), fio->old_blkaddr); > - if (!IS_ERR(mfolio)) { > - if (folio_test_uptodate(mfolio)) > - memcpy(folio_address(mfolio), > - page_address(fio->encrypted_page), PAGE_SIZE); > - f2fs_folio_put(mfolio, true); > + entry = f2fs_find_cache(META_CACHE(fio->sbi), fio->old_blkaddr); > + if (!IS_ERR(entry)) { > + f2fs_lock_cache(entry); > + if (f2fs_cache_test_uptodate(entry)) > + memcpy(cache_address(entry), page_address(fio->encrypted_page), > + F2FS_BLKSIZE); > + f2fs_put_cache(entry, true); > } > return 0; > } > @@ -3297,7 +3294,7 @@ int f2fs_do_write_data_page(struct f2fs_io_info *fio) > goto out_writepage; > } > > - /* wait for GCed page writeback via META_MAPPING */ > + /* wait for GCed page writeback via generic cache */ > if (fio->meta_gc) > f2fs_wait_on_block_writeback(inode, fio->old_blkaddr); > > @@ -4380,9 +4377,7 @@ void f2fs_invalidate_folio(struct folio *folio, size_t offset, size_t length) > return; > > if (folio_test_dirty(folio)) { > - if (inode->i_ino == F2FS_META_INO(sbi)) { > - dec_page_count(sbi, F2FS_DIRTY_META); > - } else if (inode->i_ino == F2FS_NODE_INO(sbi)) { > + if (inode->i_ino == F2FS_NODE_INO(sbi)) { > dec_page_count(sbi, F2FS_DIRTY_NODES); > } else { > inode_dec_dirty_pages(inode); > diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c > index ff379aff4472..14059a50739c 100644 > --- a/fs/f2fs/debug.c > +++ b/fs/f2fs/debug.c > @@ -224,8 +224,7 @@ static void update_general_status(struct f2fs_sb_info *sbi) > si->dirty_count = dirty_segments(sbi); > if (sbi->node_inode) > si->node_pages = NODE_MAPPING(sbi)->nrpages; > - if (sbi->meta_inode) > - si->meta_pages = META_MAPPING(sbi)->nrpages; > + si->meta_caches = META_CACHE(sbi)->num_entries; > #ifdef CONFIG_F2FS_FS_COMPRESSION > if (sbi->compress_inode) { > si->compress_pages = COMPRESS_MAPPING(sbi)->nrpages; > @@ -388,11 +387,8 @@ static void update_mem_info(struct f2fs_sb_info *sbi) > > si->page_mem += (unsigned long long)npages << PAGE_SHIFT; > } > - if (sbi->meta_inode) { > - unsigned long npages = META_MAPPING(sbi)->nrpages; > - > - si->page_mem += (unsigned long long)npages << PAGE_SHIFT; > - } > + si->page_mem += (unsigned long long)META_CACHE(sbi)->num_entries << PAGE_SHIFT; > + si->cache_mem += META_CACHE(sbi)->num_entries * sizeof(struct f2fs_cached_block); > #ifdef CONFIG_F2FS_FS_COMPRESSION > if (sbi->compress_inode) { > unsigned long npages = COMPRESS_MAPPING(sbi)->nrpages; > @@ -708,7 +704,7 @@ static int stat_show(struct seq_file *s, void *v) > seq_printf(s, " - quota data: %4d in quota files:%4d\n", > si->ndirty_qdata, si->nquota_files); > seq_printf(s, " - meta: %4d in %4d\n", > - si->ndirty_meta, si->meta_pages); > + si->ndirty_meta, si->meta_caches); > seq_printf(s, " - imeta: %4d\n", > si->ndirty_imeta); > seq_printf(s, " - fsync mark: %4lld\n", > diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h > index 6e20b3586f26..9a353dcd6658 100644 > --- a/fs/f2fs/f2fs.h > +++ b/fs/f2fs/f2fs.h > @@ -1828,7 +1828,6 @@ struct f2fs_sb_info { > struct f2fs_checkpoint *ckpt; /* raw checkpoint pointer */ > int cur_cp_pack; /* remain current cp pack */ > spinlock_t cp_lock; /* for flag in ckpt */ > - struct inode *meta_inode; /* cache meta blocks */ > struct f2fs_rwsem cp_global_sem; /* checkpoint procedure lock */ > struct f2fs_rwsem cp_rwsem; /* blocking FS operations */ > struct f2fs_rwsem node_write; /* locking node writes */ > @@ -1873,7 +1872,6 @@ struct f2fs_sb_info { > unsigned int blocksize; /* block size */ > unsigned int root_ino_num; /* root inode number*/ > unsigned int node_ino_num; /* node inode number*/ > - unsigned int meta_ino_num; /* meta inode number*/ > unsigned int log_blocks_per_seg; /* log2 blocks per segment */ > unsigned int blocks_per_seg; /* blocks per segment */ > unsigned int segs_per_sec; /* segments per section */ > @@ -2319,9 +2317,9 @@ static inline struct dirty_seglist_info *DIRTY_I(struct f2fs_sb_info *sbi) > return (struct dirty_seglist_info *)(SM_I(sbi)->dirty_info); > } > > -static inline struct address_space *META_MAPPING(struct f2fs_sb_info *sbi) > +static inline bool f2fs_is_meta_cache(struct f2fs_cached_block *entry) > { > - return sbi->meta_inode->i_mapping; > + return entry->cache && entry->cache == META_CACHE(entry->cache->sbi); > } > > static inline struct address_space *NODE_MAPPING(struct f2fs_sb_info *sbi) > @@ -2329,11 +2327,6 @@ static inline struct address_space *NODE_MAPPING(struct f2fs_sb_info *sbi) > return sbi->node_inode->i_mapping; > } > > -static inline bool is_meta_folio(struct folio *folio) > -{ > - return folio->mapping == META_MAPPING(F2FS_F_SB(folio)); > -} > - > static inline bool is_node_folio(struct folio *folio) > { > return folio->mapping == NODE_MAPPING(F2FS_F_SB(folio)); > @@ -4057,9 +4050,9 @@ bool f2fs_alloc_nid(struct f2fs_sb_info *sbi, nid_t *nid); > void f2fs_alloc_nid_done(struct f2fs_sb_info *sbi, nid_t nid); > void f2fs_alloc_nid_failed(struct f2fs_sb_info *sbi, nid_t nid); > int f2fs_try_to_free_nids(struct f2fs_sb_info *sbi, int nr_shrink); > -int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio); > -int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio); > -int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct folio *folio); > +int f2fs_recover_inline_xattr(struct inode *inode, struct f2fs_cached_block *entry); > +int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry); > +int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry); > int f2fs_restore_node_summary(struct f2fs_sb_info *sbi, > unsigned int segno, struct f2fs_summary_block *sum); > int f2fs_flush_nat_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc); > @@ -4109,11 +4102,13 @@ int f2fs_allocate_new_segments(struct f2fs_sb_info *sbi); > int f2fs_trim_fs(struct f2fs_sb_info *sbi, struct fstrim_range *range); > bool f2fs_exist_trim_candidates(struct f2fs_sb_info *sbi, > struct cp_control *cpc); > -struct folio *f2fs_get_sum_folio(struct f2fs_sb_info *sbi, unsigned int segno); > +struct f2fs_cached_block *f2fs_get_sum_cache(struct f2fs_sb_info *sbi, > + unsigned int segno); > void f2fs_update_meta_page(struct f2fs_sb_info *sbi, void *src, > block_t blk_addr); > -void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct folio *folio, > - enum iostat_type io_type); > +void f2fs_do_write_meta_cache(struct f2fs_sb_info *sbi, > + struct f2fs_cached_block *entry, > + enum iostat_type io_type); > void f2fs_do_write_node_page(unsigned int nid, struct f2fs_io_info *fio); > void f2fs_outplace_write_data(struct dnode_of_data *dn, > struct f2fs_io_info *fio); > @@ -4136,8 +4131,6 @@ void f2fs_update_device_state(struct f2fs_sb_info *sbi, nid_t ino, > block_t blkaddr, unsigned int blkcnt); > void f2fs_folio_wait_writeback(struct folio *folio, enum page_type type, > bool ordered, bool locked); > -#define f2fs_wait_on_page_writeback(page, type, ordered, locked) \ > - f2fs_folio_wait_writeback(page_folio(page), type, ordered, locked) > void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr); > void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr, > block_t len); > @@ -4203,20 +4196,21 @@ void f2fs_unlock_op(struct f2fs_sb_info *sbi, struct f2fs_lock_context *lc); > void f2fs_stop_checkpoint(struct f2fs_sb_info *sbi, bool end_io, > unsigned char reason); > void f2fs_flush_ckpt_thread(struct f2fs_sb_info *sbi); > -struct folio *f2fs_grab_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index); > -struct folio *f2fs_get_meta_folio(struct f2fs_sb_info *sbi, pgoff_t index); > -struct folio *f2fs_get_meta_folio_retry(struct f2fs_sb_info *sbi, pgoff_t index); > -struct folio *f2fs_get_tmp_folio(struct f2fs_sb_info *sbi, pgoff_t index); > +struct f2fs_cached_block *f2fs_grab_meta_cache(struct f2fs_sb_info *sbi, pgoff_t index); > +struct f2fs_cached_block *f2fs_get_meta_cache(struct f2fs_sb_info *sbi, pgoff_t index); > +struct f2fs_cached_block *f2fs_get_meta_cache_retry(struct f2fs_sb_info *sbi, pgoff_t index); > +struct f2fs_cached_block *f2fs_get_tmp_cache(struct f2fs_sb_info *sbi, pgoff_t index); > bool f2fs_is_valid_blkaddr(struct f2fs_sb_info *sbi, > block_t blkaddr, int type); > bool f2fs_is_valid_blkaddr_raw(struct f2fs_sb_info *sbi, > block_t blkaddr, int type); > -int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, > +int f2fs_ra_meta_caches(struct f2fs_sb_info *sbi, block_t start, int nrpages, > int type, bool sync); > -void f2fs_ra_meta_pages_cond(struct f2fs_sb_info *sbi, pgoff_t index, > +void f2fs_ra_meta_caches_cond(struct f2fs_sb_info *sbi, pgoff_t index, > unsigned int ra_blocks); > -long f2fs_sync_meta_pages(struct f2fs_sb_info *sbi, long nr_to_write, > - enum iostat_type io_type); > +void f2fs_write_meta_caches(struct f2fs_sb_info *sbi); > +long f2fs_sync_meta_caches(struct f2fs_sb_info *sbi, long nr_to_write, > + bool sync, enum iostat_type io_type); > void f2fs_add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type); > void f2fs_remove_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type); > void f2fs_release_ino_entry(struct f2fs_sb_info *sbi, bool all); > @@ -4403,7 +4397,7 @@ struct f2fs_stat_info { > unsigned int bimodal, avg_vblocks; > int util_free, util_valid, util_invalid; > int rsvd_segs, overp_segs; > - int dirty_count, node_pages, meta_pages, compress_pages; > + int dirty_count, node_pages, meta_caches, compress_pages; > int compress_page_hit; > int prefree_count, free_segs, free_secs; > int cp_call_count[MAX_CALL_TYPE], cp_count; > @@ -4605,7 +4599,6 @@ extern const struct file_operations f2fs_file_operations; > extern const struct inode_operations f2fs_file_inode_operations; > extern const struct address_space_operations f2fs_dblock_aops; > extern const struct address_space_operations f2fs_node_aops; > -extern const struct address_space_operations f2fs_meta_aops; > extern const struct inode_operations f2fs_dir_inode_operations; > extern const struct inode_operations f2fs_symlink_inode_operations; > extern const struct inode_operations f2fs_encrypted_symlink_inode_operations; > @@ -4626,7 +4619,7 @@ int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio); > int f2fs_convert_inline_inode(struct inode *inode); > int f2fs_try_convert_inline_dir(struct inode *dir, struct dentry *dentry); > int f2fs_write_inline_data(struct inode *inode, struct folio *folio); > -int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio); > +int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entry); > struct f2fs_dir_entry *f2fs_find_in_inline_dir(struct inode *dir, > const struct f2fs_filename *fname, struct folio **res_folio, > bool use_hash); > @@ -5186,10 +5179,8 @@ static inline void f2fs_schedule_timeout_killable(long timeout, bool io) > } > > static inline void f2fs_handle_page_eio(struct f2fs_sb_info *sbi, > - struct folio *folio, enum page_type type) > + pgoff_t ofs, enum page_type type) > { > - pgoff_t ofs = folio->index; > - > if (unlikely(f2fs_cp_error(sbi))) > return; > > @@ -5224,36 +5215,10 @@ static inline bool f2fs_is_readonly(struct f2fs_sb_info *sbi) > return f2fs_sb_has_readonly(sbi) || f2fs_readonly(sbi->sb); > } > > -static inline void f2fs_truncate_meta_inode_pages(struct f2fs_sb_info *sbi, > - block_t blkaddr, unsigned int cnt) > -{ > - bool need_submit = false; > - int i = 0; > - > - do { > - struct folio *folio; > - > - folio = filemap_get_folio(META_MAPPING(sbi), blkaddr + i); > - if (!IS_ERR(folio)) { > - if (folio_test_writeback(folio)) > - need_submit = true; > - f2fs_folio_put(folio, false); > - } > - } while (++i < cnt && !need_submit); > - > - if (need_submit) > - f2fs_submit_merged_write_cond(sbi, sbi->meta_inode, > - NULL, 0, DATA); > - > - truncate_inode_pages_range(META_MAPPING(sbi), > - F2FS_BLK_TO_BYTES((loff_t)blkaddr), > - F2FS_BLK_END_BYTES((loff_t)(blkaddr + cnt - 1))); > -} > - > static inline void f2fs_invalidate_internal_cache(struct f2fs_sb_info *sbi, > block_t blkaddr, unsigned int len) > { > - f2fs_truncate_meta_inode_pages(sbi, blkaddr, len); > + f2fs_truncate_meta_caches(sbi, blkaddr, len); > f2fs_invalidate_compress_pages_range(sbi, blkaddr, len); > } > > diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c > index a54b3ab52f1a..92daa41dd96d 100644 > --- a/fs/f2fs/file.c > +++ b/fs/f2fs/file.c > @@ -214,7 +214,7 @@ static vm_fault_t f2fs_vm_page_mkwrite(struct vm_fault *vmf) > > f2fs_folio_wait_writeback(folio, DATA, false, true); > > - /* wait for GCed page writeback via META_MAPPING */ > + /* wait for GCed page writeback via generic cache */ > f2fs_wait_on_block_writeback(inode, dn.data_blkaddr); > > /* > @@ -2563,7 +2563,7 @@ int f2fs_do_shutdown(struct f2fs_sb_info *sbi, unsigned int flag, > f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_SHUTDOWN); > break; > case F2FS_GOING_DOWN_METAFLUSH: > - f2fs_sync_meta_pages(sbi, LONG_MAX, FS_META_IO); > + f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_META_IO); > f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_SHUTDOWN); > break; > case F2FS_GOING_DOWN_NEED_FSCK: > diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c > index c4da2f31805b..54327cb2e27e 100644 > --- a/fs/f2fs/gc.c > +++ b/fs/f2fs/gc.c > @@ -1066,7 +1066,7 @@ static int gc_node_segment(struct f2fs_sb_info *sbi, > continue; > > if (phase == 0) { > - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), 1, > + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nid), 1, > META_NAT, true); > continue; > } > @@ -1217,7 +1217,8 @@ static int ra_data_block(struct inode *inode, pgoff_t index) > struct address_space *mapping = inode->i_mapping; > struct inode *atomic_inode = NULL; > struct dnode_of_data dn; > - struct folio *folio, *efolio; > + struct folio *folio; > + struct f2fs_cached_block *entry; > struct f2fs_io_info fio = { > .sbi = sbi, > .ino = inode->i_ino, > @@ -1227,6 +1228,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index) > .op_flags = 0, > .encrypted_page = NULL, > .in_list = 0, > + .is_cache = 1, > }; > int err = 0; > > @@ -1285,22 +1287,21 @@ static int ra_data_block(struct inode *inode, pgoff_t index) > > f2fs_wait_on_block_writeback(inode, dn.data_blkaddr); > > - efolio = f2fs_filemap_get_folio(META_MAPPING(sbi), dn.data_blkaddr, > - FGP_LOCK | FGP_CREAT, GFP_NOFS); > - if (IS_ERR(efolio)) { > - err = PTR_ERR(efolio); > + entry = f2fs_grab_cache(META_CACHE(sbi), dn.data_blkaddr, > + F2FS_CACHE_LOCK_CREATE); > + if (IS_ERR(entry)) { > + err = PTR_ERR(entry); > goto put_folio; > } > > - fio.encrypted_page = &efolio->page; > - > - if (folio_test_uptodate(efolio)) > - goto put_encrypted_page; > + if (f2fs_cache_test_uptodate(entry)) > + goto put_cache; > > - err = f2fs_submit_page_bio(&fio); > + fio.cache_entry = entry; > + err = f2fs_submit_cache_read(&fio); > if (err) > - goto put_encrypted_page; > - f2fs_put_page(fio.encrypted_page, false); > + goto put_cache; > + f2fs_put_cache(entry, false); > f2fs_folio_put(folio, true); > > f2fs_update_iostat(sbi, inode, FS_DATA_READ_IO, F2FS_BLKSIZE); > @@ -1309,8 +1310,8 @@ static int ra_data_block(struct inode *inode, pgoff_t index) > if (atomic_inode) > iput(atomic_inode); > return 0; > -put_encrypted_page: > - f2fs_put_page(fio.encrypted_page, true); > +put_cache: > + f2fs_put_cache(entry, true); > put_folio: > f2fs_folio_put(folio, true); > out_iput: > @@ -1320,7 +1321,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index) > } > > /* > - * Move data block via META_MAPPING while keeping locked data page. > + * Move data block via meta cache while keeping locked data page. > * This can be used to move blocks, aka LBAs, directly on disk. > */ > static int move_data_block(struct inode *inode, block_t bidx, > @@ -1337,11 +1338,13 @@ static int move_data_block(struct inode *inode, block_t bidx, > .op_flags = 0, > .encrypted_page = NULL, > .in_list = 0, > + .is_cache = 1, > }; > struct dnode_of_data dn; > struct f2fs_summary sum; > struct node_info ni; > - struct folio *folio, *mfolio, *efolio; > + struct folio *folio; > + struct f2fs_cached_block *sentry, *tentry; > block_t newaddr; > int err = 0; > bool lfs_mode = f2fs_lfs_mode(fio.sbi); > @@ -1406,20 +1409,20 @@ static int move_data_block(struct inode *inode, block_t bidx, > if (lfs_mode) > f2fs_down_write(&fio.sbi->io_order_lock); > > - mfolio = f2fs_grab_cache_folio(META_MAPPING(fio.sbi), > - fio.old_blkaddr, false); > - if (IS_ERR(mfolio)) { > - err = PTR_ERR(mfolio); > + sentry = f2fs_grab_cache(META_CACHE(fio.sbi), fio.old_blkaddr, > + F2FS_CACHE_LOCK_CREATE); > + if (IS_ERR(sentry)) { > + err = PTR_ERR(sentry); > goto up_out; > } > > - fio.encrypted_page = folio_file_page(mfolio, fio.old_blkaddr); > + fio.cache_entry = sentry; > > - /* read source block in mfolio */ > - if (!folio_test_uptodate(mfolio)) { > - err = f2fs_submit_page_bio(&fio); > + /* read source block in cache */ > + if (!f2fs_cache_test_uptodate(sentry)) { > + err = f2fs_submit_cache_read(&fio); > if (err) { > - f2fs_folio_put(mfolio, true); > + f2fs_put_cache(sentry, true); > goto up_out; > } > > @@ -1428,11 +1431,11 @@ static int move_data_block(struct inode *inode, block_t bidx, > f2fs_update_iostat(fio.sbi, NULL, FS_GDATA_READ_IO, > F2FS_BLKSIZE); > > - folio_lock(mfolio); > - if (unlikely(!is_meta_folio(mfolio) || > - !folio_test_uptodate(mfolio))) { > + f2fs_lock_cache(sentry); > + if (unlikely(!f2fs_is_meta_cache(sentry) || > + !f2fs_cache_test_uptodate(sentry))) { > err = -EIO; > - f2fs_folio_put(mfolio, true); > + f2fs_put_cache(sentry, true); > goto up_out; > } > } > @@ -1443,46 +1446,45 @@ static int move_data_block(struct inode *inode, block_t bidx, > err = f2fs_allocate_data_block(fio.sbi, NULL, fio.old_blkaddr, &newaddr, > &sum, type, NULL); > if (err) { > - f2fs_folio_put(mfolio, true); > + f2fs_put_cache(sentry, true); > /* filesystem should shutdown, no need to recovery block */ > goto up_out; > } > > - efolio = f2fs_filemap_get_folio(META_MAPPING(fio.sbi), newaddr, > - FGP_LOCK | FGP_CREAT, GFP_NOFS); > - if (IS_ERR(efolio)) { > - err = PTR_ERR(efolio); > - f2fs_folio_put(mfolio, true); > + tentry = f2fs_grab_cache(META_CACHE(fio.sbi), newaddr, > + F2FS_CACHE_LOCK_CREATE); > + if (IS_ERR(tentry)) { > + err = PTR_ERR(tentry); > + f2fs_put_cache(sentry, true); > goto recover_block; > } > > - fio.encrypted_page = &efolio->page; > + fio.cache_entry = tentry; > > /* write target block */ > - f2fs_wait_on_page_writeback(fio.encrypted_page, DATA, true, true); > - memcpy(page_address(fio.encrypted_page), > - folio_address(mfolio), PAGE_SIZE); > - f2fs_folio_put(mfolio, true); > + f2fs_cache_wait_writeback_cond(tentry, DATA); > + memcpy(cache_address(tentry), cache_address(sentry), PAGE_SIZE); > + f2fs_put_cache(sentry, true); > > f2fs_invalidate_internal_cache(fio.sbi, fio.old_blkaddr, 1); > > - set_page_dirty(fio.encrypted_page); > - if (clear_page_dirty_for_io(fio.encrypted_page)) > + f2fs_mark_cache_dirty(tentry); > + if (f2fs_clear_cache_dirty(tentry)) > dec_page_count(fio.sbi, F2FS_DIRTY_META); > > - set_page_writeback(fio.encrypted_page); > + f2fs_start_cache_writeback(tentry); > > fio.op = REQ_OP_WRITE; > fio.op_flags = REQ_SYNC; > fio.new_blkaddr = newaddr; > - f2fs_submit_page_write(&fio); > + f2fs_submit_cache_write(&fio); > > f2fs_update_iostat(fio.sbi, NULL, FS_GC_DATA_IO, F2FS_BLKSIZE); > > f2fs_update_data_blkaddr(&dn, newaddr); > set_inode_flag(inode, FI_APPEND_WRITE); > > - f2fs_put_page(fio.encrypted_page, true); > + f2fs_put_cache(tentry, true); > recover_block: > if (err) > f2fs_do_replace_block(fio.sbi, &sum, newaddr, fio.old_blkaddr, > @@ -1614,7 +1616,7 @@ static int gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum, > continue; > > if (phase == 0) { > - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), 1, > + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nid), 1, > META_NAT, true); > continue; > } > @@ -1818,28 +1820,31 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, > sum_blk_cnt = DIV_ROUND_UP(end_segno - segno, sbi->sums_per_block); > /* readahead multi ssa blocks those have contiguous address */ > if (__is_large_section(sbi)) > - f2fs_ra_meta_pages(sbi, GET_SUM_BLOCK(sbi, segno), > + f2fs_ra_meta_caches(sbi, GET_SUM_BLOCK(sbi, segno), > sum_blk_cnt, META_SSA, true); > > /* reference all summary page */ > while (segno < end_segno) { > - struct folio *sum_folio = f2fs_get_sum_folio(sbi, segno); > + struct f2fs_cached_block *sum_entry = > + f2fs_get_sum_cache(sbi, segno); > > segno += sbi->sums_per_block; > - if (IS_ERR(sum_folio)) { > - int err = PTR_ERR(sum_folio); > + if (IS_ERR(sum_entry)) { > + int err = PTR_ERR(sum_entry); > > end_segno = segno - sbi->sums_per_block; > segno = rounddown(start_segno, sbi->sums_per_block); > while (segno < end_segno) { > - sum_folio = filemap_get_folio(META_MAPPING(sbi), > + sum_entry = f2fs_find_meta_cache(sbi, > GET_SUM_BLOCK(sbi, segno)); > - folio_put_refs(sum_folio, 2); > + f2fs_put_cache(sum_entry, false); > + f2fs_put_cache(sum_entry, false); > segno += sbi->sums_per_block; > } > return err; > } > - folio_unlock(sum_folio); > + f2fs_unlock_cache(sum_entry); > + > } > > blk_start_plug(&plug); > @@ -1847,11 +1852,16 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, > segno = start_segno; > while (segno < end_segno) { > unsigned int cur_segno; > + unsigned int block_end_segno; > > /* find segment summary of victim */ > - struct folio *sum_folio = filemap_get_folio(META_MAPPING(sbi), > + struct f2fs_cached_block *sum_entry = > + f2fs_find_meta_cache(sbi, > GET_SUM_BLOCK(sbi, segno)); > - unsigned int block_end_segno = rounddown(segno, sbi->sums_per_block) > + > + f2fs_bug_on(sbi, IS_ERR(sum_entry)); > + > + block_end_segno = rounddown(segno, sbi->sums_per_block) > + sbi->sums_per_block; > > if (block_end_segno > end_segno) > @@ -1864,8 +1874,8 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, > goto next_block; > } > > - if (!folio_test_uptodate(sum_folio) || > - unlikely(f2fs_cp_error(sbi))) > + if (!f2fs_cache_test_uptodate(sum_entry) || > + unlikely(f2fs_cp_error(sbi))) > goto next_block; > > for (cur_segno = segno; cur_segno < block_end_segno; > @@ -1884,7 +1894,7 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, > data_type = (type == SUM_TYPE_DATA) ? DATA : NODE; > } > > - sum = SUM_BLK_PAGE_ADDR(sbi, sum_folio, cur_segno); > + sum = SUM_BLK_ENTRY_ADDR(sbi, sum_entry, cur_segno); > if (type != GET_SUM_TYPE(sum_footer(sbi, sum))) { > f2fs_err(sbi, "Inconsistent segment (%u) type " > "[%d, %d] in SIT and SSA", > @@ -1926,12 +1936,14 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, > cur_segno + 1 : NULL_SEGNO; > > if (unlikely(freezing(current))) { > - folio_put_refs(sum_folio, 2); > + f2fs_put_cache(sum_entry, false); > + f2fs_put_cache(sum_entry, false); > goto stop; > } > } > next_block: > - folio_put_refs(sum_folio, 2); > + f2fs_put_cache(sum_entry, false); > + f2fs_put_cache(sum_entry, false); > segno = block_end_segno; > } > > diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c > index aec06fb4fd76..2156fb1fc57d 100644 > --- a/fs/f2fs/inline.c > +++ b/fs/f2fs/inline.c > @@ -294,7 +294,7 @@ int f2fs_write_inline_data(struct inode *inode, struct folio *folio) > return 0; > } > > -int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio) > +int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entry) > { > struct f2fs_sb_info *sbi = F2FS_I_SB(inode); > struct f2fs_inode *ri = NULL; > @@ -308,12 +308,13 @@ int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio) > * x o -> remove data blocks, and then recover inline_data > * x x -> recover data blocks > */ > - if (IS_INODE(nfolio)) > - ri = F2FS_INODE(nfolio); > + if (IS_INODE(cache_folio(entry))) > + ri = &CACHED_NODE(entry)->i; > > if (f2fs_has_inline_data(inode) && > ri && (ri->i_inline & F2FS_INLINE_DATA)) { > struct folio *ifolio; > + > process_inline: > ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); > if (IS_ERR(ifolio)) > @@ -321,7 +322,7 @@ int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio) > > f2fs_folio_wait_writeback(ifolio, NODE, true, true); > > - src_addr = inline_data_addr(inode, nfolio); > + src_addr = inline_data_addr(inode, cache_folio(entry)); > dst_addr = inline_data_addr(inode, ifolio); > memcpy(dst_addr, src_addr, MAX_INLINE_DATA(inode)); > > diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c > index bac1e360d966..c533da4d4d70 100644 > --- a/fs/f2fs/inode.c > +++ b/fs/f2fs/inode.c > @@ -577,7 +577,7 @@ static int do_read_inode(struct inode *inode) > > static bool is_meta_ino(struct f2fs_sb_info *sbi, unsigned int ino) > { > - if (ino == F2FS_NODE_INO(sbi) || ino == F2FS_META_INO(sbi)) > + if (ino == F2FS_NODE_INO(sbi)) > return true; > #ifdef CONFIG_F2FS_FS_COMPRESSION > if (test_opt(sbi, COMPRESS_CACHE) && ino == F2FS_COMPRESS_INO(sbi)) > @@ -624,9 +624,6 @@ struct inode *f2fs_iget(struct super_block *sb, unsigned long ino) > if (ino == F2FS_NODE_INO(sbi)) { > inode->i_mapping->a_ops = &f2fs_node_aops; > mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS); > - } else if (ino == F2FS_META_INO(sbi)) { > - inode->i_mapping->a_ops = &f2fs_meta_aops; > - mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS); > } else if (ino == F2FS_COMPRESS_INO(sbi)) { > #ifdef CONFIG_F2FS_FS_COMPRESSION > inode->i_mapping->a_ops = &f2fs_compress_aops; > @@ -824,8 +821,7 @@ int f2fs_write_inode(struct inode *inode, struct writeback_control *wbc) > { > struct f2fs_sb_info *sbi = F2FS_I_SB(inode); > > - if (inode->i_ino == F2FS_NODE_INO(sbi) || > - inode->i_ino == F2FS_META_INO(sbi)) > + if (inode->i_ino == F2FS_NODE_INO(sbi)) > return 0; > > /* > @@ -919,7 +915,6 @@ static bool f2fs_pre_evict_inode(struct inode *inode) > f2fs_invalidate_compress_pages(sbi, inode->i_ino); > > if (inode->i_ino == F2FS_NODE_INO(sbi) || > - inode->i_ino == F2FS_META_INO(sbi) || > inode->i_ino == F2FS_COMPRESS_INO(sbi)) > return true; > > diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c > index 46bea52e35c3..398c58fc6cad 100644 > --- a/fs/f2fs/node.c > +++ b/fs/f2fs/node.c > @@ -140,38 +140,36 @@ static void clear_node_folio_dirty(struct folio *folio) > folio_clear_uptodate(folio); > } > > -static struct folio *get_current_nat_folio(struct f2fs_sb_info *sbi, nid_t nid) > +static struct f2fs_cached_block *get_current_nat_cache(struct f2fs_sb_info *sbi, > + nid_t nid) > { > - return f2fs_get_meta_folio_retry(sbi, current_nat_addr(sbi, nid)); > + return f2fs_get_meta_cache_retry(sbi, current_nat_addr(sbi, nid)); > } > > -static struct folio *get_next_nat_folio(struct f2fs_sb_info *sbi, nid_t nid) > +static struct f2fs_cached_block *get_next_nat_cache(struct f2fs_sb_info *sbi, > + nid_t nid) > { > - struct folio *src_folio; > - struct folio *dst_folio; > + struct f2fs_cached_block *src_entry; > + struct f2fs_cached_block *dst_entry; > pgoff_t dst_off; > - void *src_addr; > - void *dst_addr; > struct f2fs_nm_info *nm_i = NM_I(sbi); > > dst_off = next_nat_addr(sbi, current_nat_addr(sbi, nid)); > > /* get current nat block page with lock */ > - src_folio = get_current_nat_folio(sbi, nid); > - if (IS_ERR(src_folio)) > - return src_folio; > - dst_folio = f2fs_grab_meta_folio(sbi, dst_off); > - f2fs_bug_on(sbi, folio_test_dirty(src_folio)); > - > - src_addr = folio_address(src_folio); > - dst_addr = folio_address(dst_folio); > - memcpy(dst_addr, src_addr, PAGE_SIZE); > - folio_mark_dirty(dst_folio); > - f2fs_folio_put(src_folio, true); > + src_entry = get_current_nat_cache(sbi, nid); > + if (IS_ERR(src_entry)) > + return src_entry; > + dst_entry = f2fs_grab_meta_cache(sbi, dst_off); > + f2fs_bug_on(sbi, f2fs_cache_test_dirty(src_entry)); > + > + memcpy(cache_address(dst_entry), cache_address(src_entry), sbi->blocksize); > + f2fs_mark_cache_dirty(dst_entry); > + f2fs_put_cache(src_entry, true); > > set_to_next_nat(nm_i, nid); > > - return dst_folio; > + return dst_entry; > } > > static struct nat_entry *__alloc_nat_entry(struct f2fs_sb_info *sbi, > @@ -575,7 +573,7 @@ int f2fs_get_node_info(struct f2fs_sb_info *sbi, nid_t nid, > struct f2fs_journal *journal = curseg->journal; > nid_t start_nid = START_NID(nid); > struct f2fs_nat_block *nat_blk; > - struct folio *folio = NULL; > + struct f2fs_cached_block *entry = NULL; > struct f2fs_nat_entry ne; > struct nat_entry *e; > pgoff_t index; > @@ -629,14 +627,14 @@ int f2fs_get_node_info(struct f2fs_sb_info *sbi, nid_t nid, > index = current_nat_addr(sbi, nid); > f2fs_up_read(&nm_i->nat_tree_lock); > > - folio = f2fs_get_meta_folio(sbi, index); > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > + entry = f2fs_get_meta_cache(sbi, index); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > > - nat_blk = folio_address(folio); > + nat_blk = cache_address(entry); > ne = nat_blk->entries[nid - start_nid]; > node_info_from_raw_nat(ni, &ne); > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > sanity_check: > if (__is_valid_data_blkaddr(ni->blk_addr) && > !f2fs_is_valid_blkaddr(sbi, ni->blk_addr, > @@ -1622,7 +1620,7 @@ static struct folio *__get_node_folio(struct f2fs_sb_info *sbi, pgoff_t nid, > out_put_err: > /* ENOENT comes from read_node_folio which is not an error. */ > if (err != -ENOENT) > - f2fs_handle_page_eio(sbi, folio, NODE); > + f2fs_handle_page_eio(sbi, folio->index, NODE); > f2fs_folio_put(folio, true); > return ERR_PTR(err); > } > @@ -2598,6 +2596,7 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, > bool sync, bool mount) > { > struct f2fs_nm_info *nm_i = NM_I(sbi); > + struct f2fs_cached_block *entry = NULL; > int i = 0, ret; > nid_t nid = nm_i->next_scan_nid; > > @@ -2623,7 +2622,7 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, > } > > /* readahead nat pages to be scanned */ > - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), FREE_NID_PAGES, > + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nid), FREE_NID_PAGES, > META_NAT, true); > > f2fs_down_read(&nm_i->nat_tree_lock); > @@ -2631,14 +2630,14 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, > while (1) { > if (!test_bit_le(NAT_BLOCK_OFFSET(nid), > nm_i->nat_block_bitmap)) { > - struct folio *folio = get_current_nat_folio(sbi, nid); > + entry = get_current_nat_cache(sbi, nid); > > - if (IS_ERR(folio)) { > - ret = PTR_ERR(folio); > + if (IS_ERR(entry)) { > + ret = PTR_ERR(entry); > } else { > - ret = scan_nat_page(sbi, folio_address(folio), > + ret = scan_nat_page(sbi, cache_address(entry), > nid); > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > } > > if (ret) { > @@ -2671,7 +2670,7 @@ static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi, > > f2fs_up_read(&nm_i->nat_tree_lock); > > - f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nm_i->next_scan_nid), > + f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(nm_i->next_scan_nid), > nm_i->ra_nid_pages, META_NAT, false); > > return 0; > @@ -2826,7 +2825,7 @@ int f2fs_try_to_free_nids(struct f2fs_sb_info *sbi, int nr_shrink) > return nr - nr_shrink; > } > > -int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) > +int f2fs_recover_inline_xattr(struct inode *inode, struct f2fs_cached_block *entry) > { > void *src_addr, *dst_addr; > size_t inline_size; > @@ -2837,7 +2836,7 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) > if (IS_ERR(ifolio)) > return PTR_ERR(ifolio); > > - ri = F2FS_INODE(folio); > + ri = &CACHED_NODE(entry)->i; > if (ri->i_inline & F2FS_INLINE_XATTR) { > if (!f2fs_has_inline_xattr(inode)) { > set_inode_flag(inode, FI_INLINE_XATTR); > @@ -2852,7 +2851,7 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) > } > > dst_addr = inline_xattr_addr(inode, ifolio); > - src_addr = inline_xattr_addr(inode, folio); > + src_addr = inline_xattr_addr(inode, cache_folio(entry)); > inline_size = inline_xattr_size(inode); > > f2fs_folio_wait_writeback(ifolio, NODE, true, true); > @@ -2863,7 +2862,7 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct folio *folio) > return 0; > } > > -int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio) > +int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry) > { > struct f2fs_sb_info *sbi = F2FS_I_SB(inode); > nid_t prev_xnid = F2FS_I(inode)->i_xattr_nid; > @@ -2901,8 +2900,8 @@ int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio) > f2fs_update_inode_page(inode); > > /* 3: update and set xattr node page dirty */ > - if (folio) { > - memcpy(F2FS_NODE(xfolio), F2FS_NODE(folio), > + if (entry) { > + memcpy(F2FS_NODE(xfolio), CACHED_NODE(entry), > VALID_XATTR_BLOCK_SIZE); > folio_mark_dirty(xfolio); > } > @@ -2911,10 +2910,10 @@ int f2fs_recover_xattr_data(struct inode *inode, struct folio *folio) > return 0; > } > > -int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct folio *folio) > +int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) > { > struct f2fs_inode *src, *dst; > - nid_t ino = ino_of_node(folio); > + nid_t ino = ino_of_node(cache_folio(entry)); > struct node_info old_ni, new_ni; > struct folio *ifolio; > int err; > @@ -2940,7 +2939,7 @@ int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct folio *folio) > fill_node_footer(ifolio, ino, ino, 0, true); > set_cold_node(ifolio, false); > > - src = F2FS_INODE(folio); > + src = &CACHED_NODE(entry)->i; > dst = F2FS_INODE(ifolio); > > memcpy(dst, src, offsetof(struct f2fs_inode, i_ext)); > @@ -2999,24 +2998,24 @@ int f2fs_restore_node_summary(struct f2fs_sb_info *sbi, > nrpages = bio_max_segs(last_offset - i); > > /* readahead node pages */ > - f2fs_ra_meta_pages(sbi, addr, nrpages, META_POR, true); > + f2fs_ra_meta_caches(sbi, addr, nrpages, META_POR, true); > > for (idx = addr; idx < addr + nrpages; idx++) { > - struct folio *folio = f2fs_get_tmp_folio(sbi, idx); > + struct f2fs_cached_block *entry = > + f2fs_get_tmp_cache(sbi, idx); > > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > > - rn = F2FS_NODE(folio); > + rn = CACHED_NODE(entry); > sum_entry->nid = rn->footer.nid; > sum_entry->version = 0; > sum_entry->ofs_in_node = 0; > sum_entry++; > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > } > > - invalidate_mapping_pages(META_MAPPING(sbi), addr, > - addr + nrpages); > + f2fs_truncate_meta_caches(sbi, addr, nrpages); > } > return 0; > } > @@ -3126,7 +3125,7 @@ static int __flush_nat_entry_set(struct f2fs_sb_info *sbi, > bool to_journal = true; > struct f2fs_nat_block *nat_blk; > struct nat_entry *ne, *cur; > - struct folio *folio = NULL; > + struct f2fs_cached_block *entry = NULL; > > /* > * there are two steps to flush nat entries: > @@ -3140,11 +3139,11 @@ static int __flush_nat_entry_set(struct f2fs_sb_info *sbi, > if (to_journal) { > down_write(&curseg->journal_rwsem); > } else { > - folio = get_next_nat_folio(sbi, start_nid); > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > + entry = get_next_nat_cache(sbi, start_nid); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > > - nat_blk = folio_address(folio); > + nat_blk = cache_address(entry); > f2fs_bug_on(sbi, !nat_blk); > } > > @@ -3181,7 +3180,7 @@ static int __flush_nat_entry_set(struct f2fs_sb_info *sbi, > up_write(&curseg->journal_rwsem); > } else { > __update_nat_bits(sbi, start_nid, nat_blk); > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > } > > /* Allow dirty nats by node block allocation in write_begin */ > @@ -3252,7 +3251,7 @@ int f2fs_flush_nat_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) > __has_cursum_space(sbi, journal, > entry_count, NAT_JOURNAL)) > continue; > - f2fs_ra_meta_pages(sbi, set->set, 1, META_NAT, true); > + f2fs_ra_meta_caches(sbi, set->set, 1, META_NAT, true); > } > /* flush dirty nats in nat entry set */ > list_for_each_entry_safe(set, tmp, &sets, set_list) { > @@ -3288,15 +3287,15 @@ static int __get_nat_bitmaps(struct f2fs_sb_info *sbi) > nat_bits_addr = __start_cp_addr(sbi) + BLKS_PER_SEG(sbi) - > nm_i->nat_bits_blocks; > for (i = 0; i < nm_i->nat_bits_blocks; i++) { > - struct folio *folio; > + struct f2fs_cached_block *entry; > > - folio = f2fs_get_meta_folio(sbi, nat_bits_addr++); > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > + entry = f2fs_get_meta_cache(sbi, nat_bits_addr++); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > > memcpy(nm_i->nat_bits + F2FS_BLK_TO_BYTES(i), > - folio_address(folio), F2FS_BLKSIZE); > - f2fs_folio_put(folio, true); > + cache_address(entry), F2FS_BLKSIZE); > + f2fs_put_cache(entry, true); > } > > cp_ver |= (cur_cp_crc(ckpt) << 32); > diff --git a/fs/f2fs/node.h b/fs/f2fs/node.h > index 5e114f352099..de8dcd5d4392 100644 > --- a/fs/f2fs/node.h > +++ b/fs/f2fs/node.h > @@ -311,9 +311,9 @@ static inline void fill_node_footer_blkaddr(struct folio *folio, block_t blkaddr > rn->footer.next_blkaddr = cpu_to_le32(blkaddr); > } > > -static inline bool is_recoverable_dnode(const struct folio *folio) > +static inline bool is_recoverable_dnode(struct f2fs_sb_info *sbi, const struct folio *folio) > { > - struct f2fs_checkpoint *ckpt = F2FS_CKPT(F2FS_F_SB(folio)); > + struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); > __u64 cp_ver = cur_cp_version(ckpt); > > /* Don't care crc part, if fsck.f2fs sets it. */ > diff --git a/fs/f2fs/recovery.c b/fs/f2fs/recovery.c > index aaa5227739c8..5cb56b4c1879 100644 > --- a/fs/f2fs/recovery.c > +++ b/fs/f2fs/recovery.c > @@ -182,33 +182,33 @@ static const char *recover_printable_name(struct inode *inode, > return raw->i_name; > } > > -static int recover_dentry(struct inode *inode, struct folio *ifolio, > +static int recover_dentry(struct inode *inode, struct f2fs_cached_block *entry, > struct list_head *dir_list) > { > - struct f2fs_inode *raw_inode = F2FS_INODE(ifolio); > + struct f2fs_inode *raw_inode = &CACHED_NODE(entry)->i; > nid_t pino = le32_to_cpu(raw_inode->i_pino); > struct f2fs_dir_entry *de; > struct f2fs_filename fname; > struct qstr usr_fname; > struct folio *folio; > struct inode *dir, *einode; > - struct fsync_inode_entry *entry; > + struct fsync_inode_entry *fsync_entry; > int err = 0; > const char *name; > int name_len; > > - entry = get_fsync_inode(dir_list, pino); > - if (!entry) { > - entry = add_fsync_inode(F2FS_I_SB(inode), dir_list, > + fsync_entry = get_fsync_inode(dir_list, pino); > + if (!fsync_entry) { > + fsync_entry = add_fsync_inode(F2FS_I_SB(inode), dir_list, > pino, false); > - if (IS_ERR(entry)) { > - dir = ERR_CAST(entry); > - err = PTR_ERR(entry); > + if (IS_ERR(fsync_entry)) { > + dir = ERR_CAST(fsync_entry); > + err = PTR_ERR(fsync_entry); > goto out; > } > } > > - dir = entry->inode; > + dir = fsync_entry->inode; > err = init_recovered_filename(dir, inode, raw_inode, &fname, &usr_fname); > if (err) > goto out; > @@ -256,14 +256,14 @@ static int recover_dentry(struct inode *inode, struct folio *ifolio, > out: > name = recover_printable_name(inode, raw_inode, &name_len); > f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, dir = %llu, err = %d", > - __func__, ino_of_node(ifolio), name_len, name, > + __func__, ino_of_node(cache_folio(entry)), name_len, name, > IS_ERR(dir) ? 0 : dir->i_ino, err); > return err; > } > > -static int recover_quota_data(struct inode *inode, struct folio *folio) > +static int recover_quota_data(struct inode *inode, struct f2fs_cached_block *entry) > { > - struct f2fs_inode *raw = F2FS_INODE(folio); > + struct f2fs_inode *raw = &CACHED_NODE(entry)->i; > struct iattr attr; > uid_t i_uid = le32_to_cpu(raw->i_uid); > gid_t i_gid = le32_to_cpu(raw->i_gid); > @@ -300,9 +300,9 @@ static void recover_inline_flags(struct inode *inode, struct f2fs_inode *ri) > clear_inode_flag(inode, FI_DATA_EXIST); > } > > -static int recover_inode(struct inode *inode, struct folio *folio) > +static int recover_inode(struct inode *inode, struct f2fs_cached_block *entry) > { > - struct f2fs_inode *raw = F2FS_INODE(folio); > + struct f2fs_inode *raw = &CACHED_NODE(entry)->i; > struct f2fs_inode_info *fi = F2FS_I(inode); > const char *name; > int name_len; > @@ -310,7 +310,7 @@ static int recover_inode(struct inode *inode, struct folio *folio) > > inode->i_mode = le16_to_cpu(raw->i_mode); > > - err = recover_quota_data(inode, folio); > + err = recover_quota_data(inode, entry); > if (err) > return err; > > @@ -357,7 +357,7 @@ static int recover_inode(struct inode *inode, struct folio *folio) > name = recover_printable_name(inode, raw, &name_len); > > f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, inline = %x", > - __func__, ino_of_node(folio), name_len, name, > + __func__, ino_of_node(cache_folio(entry)), name_len, name, > raw->i_inline); > return 0; > } > @@ -386,30 +386,30 @@ static int sanity_check_node_chain(struct f2fs_sb_info *sbi, block_t blkaddr, > return 0; > > for (i = 0; i < 2; i++) { > - struct folio *folio; > + struct f2fs_cached_block *entry; > > if (!f2fs_is_valid_blkaddr(sbi, *blkaddr_fast, META_POR)) { > *is_detecting = false; > return 0; > } > > - folio = f2fs_get_tmp_folio(sbi, *blkaddr_fast); > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > + entry = f2fs_get_tmp_cache(sbi, *blkaddr_fast); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > > - if (!is_recoverable_dnode(folio)) { > - f2fs_folio_put(folio, true); > + if (!is_recoverable_dnode(sbi, cache_folio(entry))) { > + f2fs_put_cache(entry, true); > *is_detecting = false; > return 0; > } > > ra_blocks = adjust_por_ra_blocks(sbi, ra_blocks, *blkaddr_fast, > - next_blkaddr_of_node(folio)); > + next_blkaddr_of_node(cache_folio(entry))); > > - *blkaddr_fast = next_blkaddr_of_node(folio); > - f2fs_folio_put(folio, true); > + *blkaddr_fast = next_blkaddr_of_node(cache_folio(entry)); > + f2fs_put_cache(entry, true); > > - f2fs_ra_meta_pages_cond(sbi, *blkaddr_fast, ra_blocks); > + f2fs_ra_meta_caches_cond(sbi, *blkaddr_fast, ra_blocks); > } > > if (*blkaddr_fast == blkaddr) { > @@ -434,45 +434,47 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head, > blkaddr_fast = blkaddr; > > while (1) { > - struct fsync_inode_entry *entry; > - struct folio *folio; > + struct fsync_inode_entry *fsync_entry; > + struct f2fs_cached_block *entry; > + struct f2fs_node *rn; > > if (!f2fs_is_valid_blkaddr(sbi, blkaddr, META_POR)) > return 0; > > - folio = f2fs_get_tmp_folio(sbi, blkaddr); > - if (IS_ERR(folio)) { > - err = PTR_ERR(folio); > + entry = f2fs_get_tmp_cache(sbi, blkaddr); > + if (IS_ERR(entry)) { > + err = PTR_ERR(entry); > break; > } > + rn = CACHED_NODE(entry); > > - if (!is_recoverable_dnode(folio)) { > - f2fs_folio_put(folio, true); > + if (!is_recoverable_dnode(sbi, cache_folio(entry))) { > + f2fs_put_cache(entry, true); > break; > } > > - if (!is_fsync_dnode(folio)) > + if (!is_fsync_dnode(cache_folio(entry))) > goto next; > > - entry = get_fsync_inode(head, ino_of_node(folio)); > - if (!entry) { > + fsync_entry = get_fsync_inode(head, ino_of_node(cache_folio(entry))); > + if (!fsync_entry) { > bool quota_inode = false; > > if (!check_only && > - IS_INODE(folio) && > - is_dent_dnode(folio)) { > - err = f2fs_recover_inode_page(sbi, folio); > + IS_INODE(cache_folio(entry)) && > + is_dent_dnode(cache_folio(entry))) { > + err = f2fs_recover_inode_page(sbi, entry); > if (err) { > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > break; > } > quota_inode = true; > } > > - entry = add_fsync_inode(sbi, head, ino_of_node(folio), > + fsync_entry = add_fsync_inode(sbi, head, ino_of_node(cache_folio(entry)), > quota_inode); > - if (IS_ERR(entry)) { > - err = PTR_ERR(entry); > + if (IS_ERR(fsync_entry)) { > + err = PTR_ERR(fsync_entry); > /* > * CP | dnode(F) | inode(DF) > * For this case, we should not give up now. > @@ -482,18 +484,18 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head, > *new_inode = true; > goto next; > } > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > break; > } > } > - entry->blkaddr = blkaddr; > + fsync_entry->blkaddr = blkaddr; > > - if (IS_INODE(folio) && is_dent_dnode(folio)) > - entry->last_dentry = blkaddr; > + if (IS_INODE(cache_folio(entry)) && is_dent_dnode(cache_folio(entry))) > + fsync_entry->last_dentry = blkaddr; > next: > /* check next segment */ > - blkaddr = next_blkaddr_of_node(folio); > - f2fs_folio_put(folio, true); > + blkaddr = next_blkaddr_of_node(cache_folio(entry)); > + f2fs_put_cache(entry, true); > > err = sanity_check_node_chain(sbi, blkaddr, &blkaddr_fast, > &is_detecting); > @@ -519,7 +521,8 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, > unsigned short blkoff = GET_BLKOFF_FROM_SEG0(sbi, blkaddr); > struct f2fs_summary_block *sum_node; > struct f2fs_summary sum; > - struct folio *sum_folio, *node_folio; > + struct f2fs_cached_block *entry = NULL; > + struct folio *node_folio; > struct dnode_of_data tdn = *dn; > nid_t ino, nid; > struct inode *inode; > @@ -541,12 +544,12 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, > } > } > > - sum_folio = f2fs_get_sum_folio(sbi, segno); > - if (IS_ERR(sum_folio)) > - return PTR_ERR(sum_folio); > - sum_node = SUM_BLK_PAGE_ADDR(sbi, sum_folio, segno); > + entry = f2fs_get_sum_cache(sbi, segno); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > + sum_node = SUM_BLK_ENTRY_ADDR(sbi, entry, segno); > sum = sum_entries(sum_node)[blkoff]; > - f2fs_folio_put(sum_folio, true); > + f2fs_put_cache(entry, true); > got_it: > /* Use the locked dnode page and inode */ > nid = le32_to_cpu(sum.nid); > @@ -645,7 +648,7 @@ static int f2fs_reserve_new_block_retry(struct dnode_of_data *dn) > } > > static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, > - struct folio *folio) > + struct f2fs_cached_block *entry) > { > struct dnode_of_data dn; > struct node_info ni; > @@ -653,19 +656,19 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, > int err = 0, recovered = 0; > > /* step 1: recover xattr */ > - if (IS_INODE(folio)) { > - err = f2fs_recover_inline_xattr(inode, folio); > + if (IS_INODE(cache_folio(entry))) { > + err = f2fs_recover_inline_xattr(inode, entry); > if (err) > goto out; > - } else if (f2fs_has_xattr_block(ofs_of_node(folio))) { > - err = f2fs_recover_xattr_data(inode, folio); > + } else if (f2fs_has_xattr_block(ofs_of_node(cache_folio(entry)))) { > + err = f2fs_recover_xattr_data(inode, entry); > if (!err) > recovered++; > goto out; > } > > /* step 2: recover inline data */ > - err = f2fs_recover_inline_data(inode, folio); > + err = f2fs_recover_inline_data(inode, entry); > if (err) { > if (err == 1) > err = 0; > @@ -673,8 +676,8 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, > } > > /* step 3: recover data indices */ > - start = f2fs_start_bidx_of_node(ofs_of_node(folio), inode); > - end = start + ADDRS_PER_PAGE(folio, inode); > + start = f2fs_start_bidx_of_node(ofs_of_node(cache_folio(entry)), inode); > + end = start + addrs_per_page(inode, IS_INODE(cache_folio(entry))); > > set_new_dnode(&dn, inode, NULL, NULL, 0); > retry_dn: > @@ -693,12 +696,12 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, > if (err) > goto err; > > - f2fs_bug_on(sbi, ni.ino != ino_of_node(folio)); > + f2fs_bug_on(sbi, ni.ino != ino_of_node(cache_folio(entry))); > > - if (ofs_of_node(dn.node_folio) != ofs_of_node(folio)) { > + if (ofs_of_node(dn.node_folio) != ofs_of_node(cache_folio(entry))) { > f2fs_warn(sbi, "Inconsistent ofs_of_node, ino:%llu, ofs:%u, %u", > inode->i_ino, ofs_of_node(dn.node_folio), > - ofs_of_node(folio)); > + ofs_of_node(cache_folio(entry))); > err = -EFSCORRUPTED; > f2fs_handle_error(sbi, ERROR_INCONSISTENT_FOOTER); > fserror_report_file_metadata(dn.inode, err, GFP_NOFS); > @@ -709,7 +712,7 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, > block_t src, dest; > > src = f2fs_data_blkaddr(&dn); > - dest = data_blkaddr(dn.inode, folio, dn.ofs_in_node); > + dest = data_blkaddr(dn.inode, cache_folio(entry), dn.ofs_in_node); > > if (__is_valid_data_blkaddr(src) && > !f2fs_is_valid_blkaddr(sbi, src, META_POR)) { > @@ -784,16 +787,16 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, > } > } > > - copy_node_footer(dn.node_folio, folio); > + copy_node_footer(dn.node_folio, cache_folio(entry)); > fill_node_footer(dn.node_folio, dn.nid, ni.ino, > - ofs_of_node(folio), false); > + ofs_of_node(cache_folio(entry)), false); > folio_mark_dirty(dn.node_folio); > err: > f2fs_put_dnode(&dn); > out: > f2fs_notice(sbi, "recover_data: ino = %llx, nid = %x (i_size: %s), " > "range (%u, %u), recovered = %d, err = %d", > - inode->i_ino, nid_of_node(folio), > + inode->i_ino, nid_of_node(cache_folio(entry)), > file_keep_isize(inode) ? "keep" : "recover", > start, end, recovered, err); > return err; > @@ -820,26 +823,26 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list, > blkaddr = NEXT_FREE_BLKADDR(sbi, curseg); > > while (1) { > - struct fsync_inode_entry *entry; > - struct folio *folio; > + struct fsync_inode_entry *fsync_entry; > + struct f2fs_cached_block *entry; > > if (!f2fs_is_valid_blkaddr(sbi, blkaddr, META_POR)) > break; > > - folio = f2fs_get_tmp_folio(sbi, blkaddr); > - if (IS_ERR(folio)) { > - err = PTR_ERR(folio); > + entry = f2fs_get_tmp_cache(sbi, blkaddr); > + if (IS_ERR(entry)) { > + err = PTR_ERR(entry); > break; > } > > - if (!is_recoverable_dnode(folio)) { > - f2fs_folio_put(folio, true); > + if (!is_recoverable_dnode(sbi, cache_folio(entry))) { > + f2fs_put_cache(entry, true); > break; > } > recoverable_dnode++; > > - entry = get_fsync_inode(inode_list, ino_of_node(folio)); > - if (!entry) > + fsync_entry = get_fsync_inode(inode_list, ino_of_node(cache_folio(entry))); > + if (!fsync_entry) > goto next; > fsynced_dnode++; > /* > @@ -847,40 +850,40 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list, > * In this case, we can lose the latest inode(x). > * So, call recover_inode for the inode update. > */ > - if (IS_INODE(folio)) { > - err = recover_inode(entry->inode, folio); > + if (IS_INODE(cache_folio(entry))) { > + err = recover_inode(fsync_entry->inode, entry); > if (err) { > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > break; > } > recovered_inode++; > } > - if (entry->last_dentry == blkaddr) { > - err = recover_dentry(entry->inode, folio, dir_list); > + if (fsync_entry->last_dentry == blkaddr) { > + err = recover_dentry(fsync_entry->inode, entry, dir_list); > if (err) { > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > break; > } > recovered_dentry++; > } > - err = do_recover_data(sbi, entry->inode, folio); > + err = do_recover_data(sbi, fsync_entry->inode, entry); > if (err) { > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > break; > } > recovered_dnode++; > > - if (entry->blkaddr == blkaddr) > - list_move_tail(&entry->list, tmp_inode_list); > + if (fsync_entry->blkaddr == blkaddr) > + list_move_tail(&fsync_entry->list, tmp_inode_list); > next: > ra_blocks = adjust_por_ra_blocks(sbi, ra_blocks, blkaddr, > - next_blkaddr_of_node(folio)); > + next_blkaddr_of_node(cache_folio(entry))); > > /* check next segment */ > - blkaddr = next_blkaddr_of_node(folio); > - f2fs_folio_put(folio, true); > + blkaddr = next_blkaddr_of_node(cache_folio(entry)); > + f2fs_put_cache(entry, true); > > - f2fs_ra_meta_pages_cond(sbi, blkaddr, ra_blocks); > + f2fs_ra_meta_caches_cond(sbi, blkaddr, ra_blocks); > total_dnode++; > } > if (!err) > @@ -937,12 +940,11 @@ int f2fs_recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only) > destroy_fsync_dnodes(&tmp_inode_list, err); > > /* truncate meta pages to be used by the recovery */ > - truncate_inode_pages_range(META_MAPPING(sbi), > - (loff_t)MAIN_BLKADDR(sbi) << PAGE_SHIFT, -1); > - > + f2fs_truncate_meta_caches(sbi, MAIN_BLKADDR(sbi), > + MAX_BLKADDR(sbi) - MAIN_BLKADDR(sbi)); > if (err) { > truncate_inode_pages_final(NODE_MAPPING(sbi)); > - truncate_inode_pages_final(META_MAPPING(sbi)); > + f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); > } > > /* > diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c > index 56decf9c691c..cbb3a8c9f4ab 100644 > --- a/fs/f2fs/segment.c > +++ b/fs/f2fs/segment.c > @@ -2774,63 +2774,62 @@ int f2fs_npages_for_summary_flush(struct f2fs_sb_info *sbi, bool for_ra) > return 3; > } > > -/* > - * Caller should put this summary folio > - */ > -struct folio *f2fs_get_sum_folio(struct f2fs_sb_info *sbi, unsigned int segno) > + > +struct f2fs_cached_block *f2fs_get_sum_cache(struct f2fs_sb_info *sbi, > + unsigned int segno) > { > if (unlikely(f2fs_cp_error(sbi))) > return ERR_PTR(-EIO); > - return f2fs_get_meta_folio_retry(sbi, GET_SUM_BLOCK(sbi, segno)); > + return f2fs_get_meta_cache_retry(sbi, GET_SUM_BLOCK(sbi, segno)); > } > > void f2fs_update_meta_page(struct f2fs_sb_info *sbi, > void *src, block_t blk_addr) > { > - struct folio *folio; > + struct f2fs_cached_block *entry; > > if (!f2fs_sb_has_packed_ssa(sbi)) > - folio = f2fs_grab_meta_folio(sbi, blk_addr); > + entry = f2fs_grab_meta_cache(sbi, blk_addr); > else > - folio = f2fs_get_meta_folio_retry(sbi, blk_addr); > + entry = f2fs_get_meta_cache_retry(sbi, blk_addr); > > - if (IS_ERR(folio)) > + if (IS_ERR(entry)) > return; > > - memcpy(folio_address(folio), src, PAGE_SIZE); > - folio_mark_dirty(folio); > - f2fs_folio_put(folio, true); > + memcpy(cache_address(entry), src, sbi->blocksize); > + f2fs_mark_cache_dirty(entry); > + f2fs_put_cache(entry, true); > } > > static void write_sum_page(struct f2fs_sb_info *sbi, > struct f2fs_summary_block *sum_blk, unsigned int segno) > { > - struct folio *folio; > + struct f2fs_cached_block *entry; > > if (!f2fs_sb_has_packed_ssa(sbi)) > return f2fs_update_meta_page(sbi, (void *)sum_blk, > GET_SUM_BLOCK(sbi, segno)); > > - folio = f2fs_get_sum_folio(sbi, segno); > - if (IS_ERR(folio)) > + entry = f2fs_get_sum_cache(sbi, GET_SUM_BLOCK(sbi, segno)); > + if (IS_ERR(entry)) > return; > > - memcpy(SUM_BLK_PAGE_ADDR(sbi, folio, segno), sum_blk, > + memcpy(SUM_BLK_ENTRY_ADDR(sbi, entry, segno), sum_blk, > sbi->sum_blocksize); > - folio_mark_dirty(folio); > - f2fs_folio_put(folio, true); > + f2fs_mark_cache_dirty(entry); > + f2fs_put_cache(entry, true); > } > > static void write_current_sum_page(struct f2fs_sb_info *sbi, > int type, block_t blk_addr) > { > struct curseg_info *curseg = CURSEG_I(sbi, type); > - struct folio *folio = f2fs_grab_meta_folio(sbi, blk_addr); > + struct f2fs_cached_block *entry = f2fs_grab_meta_cache(sbi, blk_addr); > struct f2fs_summary_block *src = curseg->sum_blk; > struct f2fs_summary_block *dst; > > - dst = folio_address(folio); > - memset(dst, 0, PAGE_SIZE); > + dst = cache_address(entry); > + memset(dst, 0, sbi->blocksize); > > mutex_lock(&curseg->curseg_mutex); > > @@ -2843,8 +2842,8 @@ static void write_current_sum_page(struct f2fs_sb_info *sbi, > > mutex_unlock(&curseg->curseg_mutex); > > - folio_mark_dirty(folio); > - f2fs_folio_put(folio, true); > + f2fs_mark_cache_dirty(entry); > + f2fs_put_cache(entry, true); > } > > static int is_next_segment_free(struct f2fs_sb_info *sbi, > @@ -3160,7 +3159,7 @@ static int change_curseg(struct f2fs_sb_info *sbi, int type) > struct curseg_info *curseg = CURSEG_I(sbi, type); > unsigned int new_segno = curseg->next_segno; > struct f2fs_summary_block *sum_node; > - struct folio *sum_folio; > + struct f2fs_cached_block *entry = NULL; > > if (curseg->inited) > write_sum_page(sbi, curseg->sum_blk, curseg->segno); > @@ -3176,15 +3175,15 @@ static int change_curseg(struct f2fs_sb_info *sbi, int type) > curseg->alloc_type = SSR; > curseg->next_blkoff = __next_free_blkoff(sbi, curseg->segno, 0); > > - sum_folio = f2fs_get_sum_folio(sbi, new_segno); > - if (IS_ERR(sum_folio)) { > + entry = f2fs_get_sum_cache(sbi, new_segno); > + if (IS_ERR(entry)) { > /* GC won't be able to use stale summary pages by cp_error */ > memset(curseg->sum_blk, 0, sbi->sum_entry_size); > - return PTR_ERR(sum_folio); > + return PTR_ERR(entry); > } > - sum_node = SUM_BLK_PAGE_ADDR(sbi, sum_folio, new_segno); > + sum_node = SUM_BLK_ENTRY_ADDR(sbi, entry, new_segno); > memcpy(curseg->sum_blk, sum_node, sbi->sum_entry_size); > - f2fs_folio_put(sum_folio, true); > + f2fs_put_cache(entry, true); > return 0; > } > > @@ -4127,8 +4126,9 @@ static void do_write_page(struct f2fs_summary *sum, struct f2fs_io_info *fio) > f2fs_up_read(&fio->sbi->io_order_lock); > } > > -void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct folio *folio, > - enum iostat_type io_type) > +void f2fs_do_write_meta_cache(struct f2fs_sb_info *sbi, > + struct f2fs_cached_block *entry, > + enum iostat_type io_type) > { > struct f2fs_io_info fio = { > .sbi = sbi, > @@ -4136,20 +4136,21 @@ void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct folio *folio, > .temp = HOT, > .op = REQ_OP_WRITE, > .op_flags = REQ_SYNC | REQ_META | REQ_PRIO, > - .old_blkaddr = folio->index, > - .new_blkaddr = folio->index, > - .folio = folio, > + .old_blkaddr = entry->index, > + .new_blkaddr = entry->index, > .encrypted_page = NULL, > .in_list = 0, > + .cache_entry = entry, > + .is_cache = 1, > }; > > - if (unlikely(folio->index >= MAIN_BLKADDR(sbi))) > + if (unlikely(entry->index >= MAIN_BLKADDR(sbi))) > fio.op_flags &= ~REQ_META; > > - folio_start_writeback(folio); > - f2fs_submit_page_write(&fio); > + f2fs_start_cache_writeback(entry); > + f2fs_submit_cache_write(&fio); > > - stat_inc_meta_count(sbi, folio->index); > + stat_inc_meta_count(sbi, entry->index); > f2fs_update_iostat(sbi, NULL, io_type, F2FS_BLKSIZE); > } > > @@ -4206,7 +4207,7 @@ int f2fs_inplace_write_data(struct f2fs_io_info *fio) > } > > if (fio->meta_gc) > - f2fs_truncate_meta_inode_pages(sbi, fio->new_blkaddr, 1); > + f2fs_truncate_meta_caches(sbi, fio->new_blkaddr, 1); > > stat_inc_inplace_blocks(fio->sbi); > > @@ -4372,7 +4373,7 @@ void f2fs_folio_wait_writeback(struct folio *folio, enum page_type type, > void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr) > { > struct f2fs_sb_info *sbi = F2FS_I_SB(inode); > - struct folio *cfolio; > + struct f2fs_cached_block *entry; > > if (!f2fs_meta_inode_gc_required(inode)) > return; > @@ -4380,11 +4381,12 @@ void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr) > if (!__is_valid_data_blkaddr(blkaddr)) > return; > > - cfolio = filemap_lock_folio(META_MAPPING(sbi), blkaddr); > - if (!IS_ERR(cfolio)) { > - f2fs_folio_wait_writeback(cfolio, DATA, true, true); > - f2fs_folio_put(cfolio, true); > - } > + entry = f2fs_find_cache(META_CACHE(sbi), blkaddr); > + if (IS_ERR(entry)) > + return; > + f2fs_lock_cache(entry); > + f2fs_cache_wait_writeback_cond(entry, DATA); > + f2fs_put_cache(entry, true); > } > > void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr, > @@ -4399,7 +4401,7 @@ void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr, > for (i = 0; i < len; i++) > f2fs_wait_on_block_writeback(inode, blkaddr + i); > > - f2fs_truncate_meta_inode_pages(sbi, blkaddr, len); > + f2fs_truncate_meta_caches(sbi, blkaddr, len); > } > > static int read_compacted_summaries(struct f2fs_sb_info *sbi) > @@ -4407,16 +4409,16 @@ static int read_compacted_summaries(struct f2fs_sb_info *sbi) > struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); > struct curseg_info *seg_i; > unsigned char *kaddr; > - struct folio *folio; > + struct f2fs_cached_block *entry; > block_t start; > int i, j, offset; > > start = start_sum_block(sbi); > > - folio = f2fs_get_meta_folio(sbi, start++); > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > - kaddr = folio_address(folio); > + entry = f2fs_get_meta_cache(sbi, start++); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > + kaddr = cache_address(entry); > > /* Step 1: restore nat cache */ > seg_i = CURSEG_I(sbi, CURSEG_HOT_DATA); > @@ -4453,16 +4455,16 @@ static int read_compacted_summaries(struct f2fs_sb_info *sbi) > SUM_FOOTER_SIZE) > continue; > > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > > - folio = f2fs_get_meta_folio(sbi, start++); > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > - kaddr = folio_address(folio); > + entry = f2fs_get_meta_cache(sbi, start++); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > + kaddr = cache_address(entry); > offset = 0; > } > } > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > return 0; > } > > @@ -4471,7 +4473,7 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type) > struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); > struct f2fs_summary_block *sum; > struct curseg_info *curseg; > - struct folio *new; > + struct f2fs_cached_block *entry; > unsigned short blk_off; > unsigned int segno = 0; > block_t blk_addr = 0; > @@ -4498,10 +4500,10 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type) > blk_addr = GET_SUM_BLOCK(sbi, segno); > } > > - new = f2fs_get_meta_folio(sbi, blk_addr); > - if (IS_ERR(new)) > - return PTR_ERR(new); > - sum = folio_address(new); > + entry = f2fs_get_meta_cache(sbi, blk_addr); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > + sum = cache_address(entry); > > if (IS_NODESEG(type)) { > if (__exist_node_summaries(sbi)) { > @@ -4538,7 +4540,7 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type) > curseg->next_blkoff = blk_off; > mutex_unlock(&curseg->curseg_mutex); > out: > - f2fs_folio_put(new, true); > + f2fs_put_cache(entry, true); > return err; > } > > @@ -4553,8 +4555,8 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi) > int npages = f2fs_npages_for_summary_flush(sbi, true); > > if (npages >= 2) > - f2fs_ra_meta_pages(sbi, start_sum_block(sbi), npages, > - META_CP, true); > + f2fs_ra_meta_caches(sbi, start_sum_block(sbi), > + npages, META_CP, true); > > /* restore for compacted data summary */ > err = read_compacted_summaries(sbi); > @@ -4564,7 +4566,7 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi) > } > > if (__exist_node_summaries(sbi)) > - f2fs_ra_meta_pages(sbi, > + f2fs_ra_meta_caches(sbi, > sum_blk_addr(sbi, NR_CURSEG_PERSIST_TYPE, type), > NR_CURSEG_PERSIST_TYPE - type, META_CP, true); > > @@ -4587,16 +4589,16 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi) > > static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr) > { > - struct folio *folio; > + struct f2fs_cached_block *entry = NULL; > unsigned char *kaddr; > struct f2fs_summary *summary; > struct curseg_info *seg_i; > int written_size = 0; > int i, j; > > - folio = f2fs_grab_meta_folio(sbi, blkaddr++); > - kaddr = folio_address(folio); > - memset(kaddr, 0, PAGE_SIZE); > + entry = f2fs_grab_meta_cache(sbi, blkaddr++); > + kaddr = cache_address(entry); > + memset(kaddr, 0, sbi->blocksize); > > /* Step 1: write nat cache */ > seg_i = CURSEG_I(sbi, CURSEG_HOT_DATA); > @@ -4612,10 +4614,10 @@ static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr) > for (i = CURSEG_HOT_DATA; i <= CURSEG_COLD_DATA; i++) { > seg_i = CURSEG_I(sbi, i); > for (j = 0; j < f2fs_curseg_valid_blocks(sbi, i); j++) { > - if (!folio) { > - folio = f2fs_grab_meta_folio(sbi, blkaddr++); > - kaddr = folio_address(folio); > - memset(kaddr, 0, PAGE_SIZE); > + if (!entry) { > + entry = f2fs_grab_meta_cache(sbi, blkaddr++); > + kaddr = cache_address(entry); > + memset(kaddr, 0, sbi->blocksize); > written_size = 0; > } > summary = (struct f2fs_summary *)(kaddr + written_size); > @@ -4626,14 +4628,14 @@ static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr) > SUM_FOOTER_SIZE) > continue; > > - folio_mark_dirty(folio); > - f2fs_folio_put(folio, true); > - folio = NULL; > + f2fs_mark_cache_dirty(entry); > + f2fs_put_cache(entry, true); > + entry = NULL; > } > } > - if (folio) { > - folio_mark_dirty(folio); > - f2fs_folio_put(folio, true); > + if (entry) { > + f2fs_mark_cache_dirty(entry); > + f2fs_put_cache(entry, true); > } > } > > @@ -4687,29 +4689,29 @@ int f2fs_lookup_journal_in_cursum(struct f2fs_sb_info *sbi, > return -1; > } > > -static struct folio *get_current_sit_folio(struct f2fs_sb_info *sbi, > +static struct f2fs_cached_block *get_current_sit_cache(struct f2fs_sb_info *sbi, > unsigned int segno) > { > - return f2fs_get_meta_folio(sbi, current_sit_addr(sbi, segno)); > + return f2fs_get_meta_cache(sbi, current_sit_addr(sbi, segno)); > } > > -static struct folio *get_next_sit_folio(struct f2fs_sb_info *sbi, > +static struct f2fs_cached_block *get_next_sit_cache(struct f2fs_sb_info *sbi, > unsigned int start) > { > struct sit_info *sit_i = SIT_I(sbi); > - struct folio *folio; > + struct f2fs_cached_block *entry; > pgoff_t src_off, dst_off; > > src_off = current_sit_addr(sbi, start); > dst_off = next_sit_addr(sbi, src_off); > > - folio = f2fs_grab_meta_folio(sbi, dst_off); > - seg_info_to_sit_folio(sbi, folio, start); > + entry = f2fs_grab_meta_cache(sbi, dst_off); > + seg_info_to_sit_block(sbi, entry, start); > > - folio_mark_dirty(folio); > + f2fs_mark_cache_dirty(entry); > set_to_next_sit(sit_i, start); > > - return folio; > + return entry; > } > > static struct sit_entry_set *grab_sit_entry_set(void) > @@ -4839,8 +4841,9 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) > * #2, flush sit entries to sit page. > */ > list_for_each_entry_safe(ses, tmp, head, set_list) { > - struct folio *folio = NULL; > + struct f2fs_cached_block *entry = NULL; > struct f2fs_sit_block *raw_sit = NULL; > + > unsigned int start_segno = ses->start_segno; > unsigned int end = min(start_segno + SIT_ENTRY_PER_BLOCK, > (unsigned long)MAIN_SEGS(sbi)); > @@ -4854,8 +4857,8 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) > if (to_journal) { > down_write(&curseg->journal_rwsem); > } else { > - folio = get_next_sit_folio(sbi, start_segno); > - raw_sit = folio_address(folio); > + entry = get_next_sit_cache(sbi, start_segno); > + raw_sit = cache_address(entry); > } > > /* flush dirty sit entries in region of current sit set */ > @@ -4900,7 +4903,7 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc) > if (to_journal) > up_write(&curseg->journal_rwsem); > else > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > > f2fs_bug_on(sbi, ses->entry_cnt); > release_sit_entry_set(ses); > @@ -5091,7 +5094,7 @@ static int build_sit_entries(struct f2fs_sb_info *sbi) > block_t sit_valid_blocks[2] = {0, 0}; > > do { > - readed = f2fs_ra_meta_pages(sbi, start_blk, BIO_MAX_VECS, > + readed = f2fs_ra_meta_caches(sbi, start_blk, BIO_MAX_VECS, > META_SIT, true); > > start = start_blk * sit_i->sents_per_block; > @@ -5099,15 +5102,15 @@ static int build_sit_entries(struct f2fs_sb_info *sbi) > > for (; start < end && start < MAIN_SEGS(sbi); start++) { > struct f2fs_sit_block *sit_blk; > - struct folio *folio; > + struct f2fs_cached_block *entry; > > se = &sit_i->sentries[start]; > - folio = get_current_sit_folio(sbi, start); > - if (IS_ERR(folio)) > - return PTR_ERR(folio); > - sit_blk = folio_address(folio); > + entry = get_current_sit_cache(sbi, start); > + if (IS_ERR(entry)) > + return PTR_ERR(entry); > + sit_blk = cache_address(entry); > sit = sit_blk->entries[SIT_ENTRY_OFFSET(sit_i, start)]; > - f2fs_folio_put(folio, true); > + f2fs_put_cache(entry, true); > > err = check_block_count(sbi, start, &sit); > if (err) > diff --git a/fs/f2fs/segment.h b/fs/f2fs/segment.h > index db1079169a23..fff35c63a00f 100644 > --- a/fs/f2fs/segment.h > +++ b/fs/f2fs/segment.h > @@ -95,6 +95,8 @@ static inline void sanity_check_seg_type(struct f2fs_sb_info *sbi, > #define GET_SUM_BLKOFF(sbi, segno) (segno % (sbi)->sums_per_block) > #define SUM_BLK_PAGE_ADDR(sbi, folio, segno) \ > (folio_address(folio) + GET_SUM_BLKOFF(sbi, segno) * (sbi)->sum_blocksize) > +#define SUM_BLK_ENTRY_ADDR(sbi, entry, segno) \ > + (cache_address(entry) + GET_SUM_BLKOFF(sbi, segno) * (sbi)->sum_blocksize) > > #define GET_SUM_TYPE(footer) ((footer)->entry_type) > #define SET_SUM_TYPE(footer, type) ((footer)->entry_type = (type)) > @@ -417,8 +419,8 @@ static inline void __seg_info_to_raw_sit(struct seg_entry *se, > rs->mtime = cpu_to_le64(se->mtime); > } > > -static inline void seg_info_to_sit_folio(struct f2fs_sb_info *sbi, > - struct folio *folio, unsigned int start) > +static inline void seg_info_to_sit_block(struct f2fs_sb_info *sbi, > + struct f2fs_cached_block *entry, unsigned int start) > { > struct f2fs_sit_block *raw_sit; > struct seg_entry *se; > @@ -427,8 +429,8 @@ static inline void seg_info_to_sit_folio(struct f2fs_sb_info *sbi, > (unsigned long)MAIN_SEGS(sbi)); > int i; > > - raw_sit = folio_address(folio); > - memset(raw_sit, 0, PAGE_SIZE); > + raw_sit = cache_address(entry); > + memset(raw_sit, 0, sbi->blocksize); > for (i = 0; i < end - start; i++) { > rs = &raw_sit->entries[i]; > se = get_seg_entry(sbi, start + i); > @@ -991,6 +993,27 @@ static inline int nr_pages_to_skip(struct f2fs_sb_info *sbi, int type) > return 0; > } > > +/* > + * When writing cache asynchronously, align nr_to_write to BIO_MAX_VECS. > + */ > +static inline long adjust_flush_cache_number(struct f2fs_sb_info *sbi, int type) > +{ > + long nr_to_write; > + > + switch (type) { > + case META: > + nr_to_write = BIO_MAX_VECS; > + break; > + case NODE: > + nr_to_write = BIO_MAX_VECS << 1; > + break; > + default: > + f2fs_bug_on(sbi, 1); > + return 0; > + } > + return nr_to_write; > +} > + > /* > * When writing pages, it'd better align nr_to_write for segment size. > */ > diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c > index 89affe72f4fc..d3dee9b7d999 100644 > --- a/fs/f2fs/super.c > +++ b/fs/f2fs/super.c > @@ -1839,8 +1839,7 @@ static int f2fs_drop_inode(struct inode *inode) > * drop useless meta/node dirty pages. > */ > if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) { > - if (inode->i_ino == F2FS_NODE_INO(sbi) || > - inode->i_ino == F2FS_META_INO(sbi)) { > + if (inode->i_ino == F2FS_NODE_INO(sbi)) { > trace_f2fs_drop_inode(inode, 1); > return 1; > } > @@ -1942,8 +1941,7 @@ static void f2fs_dirty_inode(struct inode *inode, int flags) > { > struct f2fs_sb_info *sbi = F2FS_I_SB(inode); > > - if (inode->i_ino == F2FS_NODE_INO(sbi) || > - inode->i_ino == F2FS_META_INO(sbi)) > + if (inode->i_ino == F2FS_NODE_INO(sbi)) > return; > > if (is_inode_flag_set(inode, FI_AUTO_RECOVER)) > @@ -2039,9 +2037,10 @@ static void f2fs_put_super(struct super_block *sb) > > f2fs_wait_on_all_pages(sbi, F2FS_WB_CP_DATA); > > - if (err || f2fs_cp_error(sbi)) { > + if (err || f2fs_cp_error(sbi) || > + unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) { > truncate_inode_pages_final(NODE_MAPPING(sbi)); > - truncate_inode_pages_final(META_MAPPING(sbi)); > + f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); > } > > f2fs_bug_on(sbi, sbi->fsync_node_num); > @@ -2051,9 +2050,6 @@ static void f2fs_put_super(struct super_block *sb) > iput(sbi->node_inode); > sbi->node_inode = NULL; > > - iput(sbi->meta_inode); > - sbi->meta_inode = NULL; > - > f2fs_destroy_cache(META_CACHE(sbi)); > > /* Should check the page counts after dropping all node/meta pages */ > @@ -4383,7 +4379,6 @@ static void init_sb_info(struct f2fs_sb_info *sbi) > sbi->allocate_section_policy = ALLOCATE_FORWARD_NOHINT; > F2FS_ROOT_INO(sbi) = le32_to_cpu(raw_super->root_ino); > F2FS_NODE_INO(sbi) = le32_to_cpu(raw_super->node_ino); > - F2FS_META_INO(sbi) = le32_to_cpu(raw_super->meta_ino); > sbi->cur_victim_sec = NULL_SECNO; > sbi->gc_mode = GC_NORMAL; > sbi->next_victim_seg[BG_GC] = NULL_SEGNO; > @@ -5224,18 +5219,10 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) > if (err) > goto free_page_array_cache; > > - /* get an inode for meta space */ > - sbi->meta_inode = f2fs_iget(sb, F2FS_META_INO(sbi)); > - if (IS_ERR(sbi->meta_inode)) { > - f2fs_err(sbi, "Failed to read F2FS meta data inode"); > - err = PTR_ERR(sbi->meta_inode); > - goto free_meta_cache; > - } > - > err = f2fs_get_valid_checkpoint(sbi); > if (err) { > f2fs_err(sbi, "Failed to get valid F2FS checkpoint"); > - goto free_meta_inode; > + goto free_meta_cache; > } > > if (__is_set_ckpt_flags(F2FS_CKPT(sbi), CP_QUOTA_NEED_FSCK_FLAG)) > @@ -5529,7 +5516,7 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) > * followed by f2fs_write_checkpoint() through f2fs_write_node_pages(), which > * falls into an infinite loop in f2fs_sync_meta_pages(). > */ > - truncate_inode_pages_final(META_MAPPING(sbi)); > + f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); > /* evict some inodes being cached by GC */ > evict_inodes(sb); > f2fs_unregister_sysfs(sbi); > @@ -5559,10 +5546,6 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) > free_devices: > destroy_device_list(sbi); > kvfree(sbi->ckpt); > -free_meta_inode: > - make_bad_inode(sbi->meta_inode); > - iput(sbi->meta_inode); > - sbi->meta_inode = NULL; > free_meta_cache: > f2fs_destroy_cache(META_CACHE(sbi)); > free_page_array_cache: > diff --git a/include/linux/f2fs_fs.h b/include/linux/f2fs_fs.h > index bb2b6cd5d507..0c027d00a1ea 100644 > --- a/include/linux/f2fs_fs.h > +++ b/include/linux/f2fs_fs.h > @@ -35,7 +35,6 @@ > > #define F2FS_ROOT_INO(sbi) ((sbi)->root_ino_num) > #define F2FS_NODE_INO(sbi) ((sbi)->node_ino_num) > -#define F2FS_META_INO(sbi) ((sbi)->meta_ino_num) > #define F2FS_COMPRESS_INO(sbi) (NM_I(sbi)->max_nid) > > #define F2FS_MAX_QUOTAS 3 > -- > 2.49.0 > > > > _______________________________________________ > Linux-f2fs-devel mailing list > Linux-f2fs-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH v1 06/12] f2fs: cache: initialize node cache 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel -1 siblings, 0 replies; 34+ messages in thread From: Chao Yu @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-f2fs-devel, linux-kernel, Chao Yu This patch introduces node_blocks in f2fs_sb_info structure, initializes and destroys the node cache during filesystem mount and unmount. It also introduces helper wrappers for node cache operation, and registers node cache into the memory shrinker and writeback kthread. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 9 ++++++++- fs/f2fs/cache.h | 11 +++++++++++ fs/f2fs/f2fs.h | 1 + fs/f2fs/super.c | 9 ++++++++- 4 files changed, 28 insertions(+), 2 deletions(-) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index afef808e485a..6c9b7a6d2313 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -609,7 +609,14 @@ static unsigned long f2fs_do_shrink_cache(struct f2fs_cached_block_list *cache, unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, unsigned long nr_to_scan) { - return f2fs_do_shrink_cache(META_CACHE(sbi), nr_to_scan); + unsigned long freed; + + freed = f2fs_do_shrink_cache(META_CACHE(sbi), nr_to_scan); + if (freed >= nr_to_scan) + return freed; + + freed += f2fs_do_shrink_cache(NODE_CACHE(sbi), nr_to_scan - freed); + return freed; } static int f2fs_cache_writeback_kthread(void *data) diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index 5e9fe8c0b15c..d843b9caba47 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -180,6 +180,7 @@ int f2fs_start_cache_wb_thread(struct f2fs_sb_info *sbi); void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); #define META_CACHE(sbi) (&(sbi)->meta_blocks) +#define NODE_CACHE(sbi) (&(sbi)->node_blocks) #define f2fs_find_meta_cache(sbi, blkaddr) \ f2fs_find_cache(META_CACHE(sbi), blkaddr) @@ -188,6 +189,16 @@ void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); #define f2fs_truncate_meta_caches(sbi, start, len) \ f2fs_drop_cache_range(META_CACHE(sbi), start, len, true) +#define f2fs_grab_node_cache(sbi, blkaddr) \ + f2fs_grab_cache(NODE_CACHE(sbi), blkaddr, \ + F2FS_CACHE_LOCK_CREATE) +#define f2fs_find_node_cache(sbi, blkaddr) \ + f2fs_find_cache(NODE_CACHE(sbi), blkaddr) +#define f2fs_invalidate_node_cache(sbi, blkaddr) \ + f2fs_drop_cache_range(NODE_CACHE(sbi), blkaddr, 1, false) +#define f2fs_truncate_node_caches(sbi, start, len) \ + f2fs_drop_cache_range(NODE_CACHE(sbi), start, len, true) + unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, unsigned long nr_to_scan); diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 9a353dcd6658..4a811e9d325b 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -2106,6 +2106,7 @@ struct f2fs_sb_info { /* f2fs internal cache */ struct f2fs_cached_block_list meta_blocks; + struct f2fs_cached_block_list node_blocks; /* internal cache flush thread */ struct f2fs_cache_kthread cache_thread; diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index d3dee9b7d999..83496c46c89f 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -2051,6 +2051,7 @@ static void f2fs_put_super(struct super_block *sb) sbi->node_inode = NULL; f2fs_destroy_cache(META_CACHE(sbi)); + f2fs_destroy_cache(NODE_CACHE(sbi)); /* Should check the page counts after dropping all node/meta pages */ for (i = 0; i < NR_COUNT_TYPE; i++) { @@ -5219,10 +5220,14 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) if (err) goto free_page_array_cache; + err = f2fs_init_cache(sbi, NODE_CACHE(sbi), F2FS_NODE_CACHE); + if (err) + goto free_meta_cache; + err = f2fs_get_valid_checkpoint(sbi); if (err) { f2fs_err(sbi, "Failed to get valid F2FS checkpoint"); - goto free_meta_cache; + goto free_node_cache; } if (__is_set_ckpt_flags(F2FS_CKPT(sbi), CP_QUOTA_NEED_FSCK_FLAG)) @@ -5546,6 +5551,8 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) free_devices: destroy_device_list(sbi); kvfree(sbi->ckpt); +free_node_cache: + f2fs_destroy_cache(NODE_CACHE(sbi)); free_meta_cache: f2fs_destroy_cache(META_CACHE(sbi)); free_page_array_cache: -- 2.49.0 ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [f2fs-dev] [PATCH v1 06/12] f2fs: cache: initialize node cache @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-kernel, linux-f2fs-devel This patch introduces node_blocks in f2fs_sb_info structure, initializes and destroys the node cache during filesystem mount and unmount. It also introduces helper wrappers for node cache operation, and registers node cache into the memory shrinker and writeback kthread. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 9 ++++++++- fs/f2fs/cache.h | 11 +++++++++++ fs/f2fs/f2fs.h | 1 + fs/f2fs/super.c | 9 ++++++++- 4 files changed, 28 insertions(+), 2 deletions(-) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index afef808e485a..6c9b7a6d2313 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -609,7 +609,14 @@ static unsigned long f2fs_do_shrink_cache(struct f2fs_cached_block_list *cache, unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, unsigned long nr_to_scan) { - return f2fs_do_shrink_cache(META_CACHE(sbi), nr_to_scan); + unsigned long freed; + + freed = f2fs_do_shrink_cache(META_CACHE(sbi), nr_to_scan); + if (freed >= nr_to_scan) + return freed; + + freed += f2fs_do_shrink_cache(NODE_CACHE(sbi), nr_to_scan - freed); + return freed; } static int f2fs_cache_writeback_kthread(void *data) diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index 5e9fe8c0b15c..d843b9caba47 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -180,6 +180,7 @@ int f2fs_start_cache_wb_thread(struct f2fs_sb_info *sbi); void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); #define META_CACHE(sbi) (&(sbi)->meta_blocks) +#define NODE_CACHE(sbi) (&(sbi)->node_blocks) #define f2fs_find_meta_cache(sbi, blkaddr) \ f2fs_find_cache(META_CACHE(sbi), blkaddr) @@ -188,6 +189,16 @@ void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); #define f2fs_truncate_meta_caches(sbi, start, len) \ f2fs_drop_cache_range(META_CACHE(sbi), start, len, true) +#define f2fs_grab_node_cache(sbi, blkaddr) \ + f2fs_grab_cache(NODE_CACHE(sbi), blkaddr, \ + F2FS_CACHE_LOCK_CREATE) +#define f2fs_find_node_cache(sbi, blkaddr) \ + f2fs_find_cache(NODE_CACHE(sbi), blkaddr) +#define f2fs_invalidate_node_cache(sbi, blkaddr) \ + f2fs_drop_cache_range(NODE_CACHE(sbi), blkaddr, 1, false) +#define f2fs_truncate_node_caches(sbi, start, len) \ + f2fs_drop_cache_range(NODE_CACHE(sbi), start, len, true) + unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, unsigned long nr_to_scan); diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 9a353dcd6658..4a811e9d325b 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -2106,6 +2106,7 @@ struct f2fs_sb_info { /* f2fs internal cache */ struct f2fs_cached_block_list meta_blocks; + struct f2fs_cached_block_list node_blocks; /* internal cache flush thread */ struct f2fs_cache_kthread cache_thread; diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index d3dee9b7d999..83496c46c89f 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -2051,6 +2051,7 @@ static void f2fs_put_super(struct super_block *sb) sbi->node_inode = NULL; f2fs_destroy_cache(META_CACHE(sbi)); + f2fs_destroy_cache(NODE_CACHE(sbi)); /* Should check the page counts after dropping all node/meta pages */ for (i = 0; i < NR_COUNT_TYPE; i++) { @@ -5219,10 +5220,14 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) if (err) goto free_page_array_cache; + err = f2fs_init_cache(sbi, NODE_CACHE(sbi), F2FS_NODE_CACHE); + if (err) + goto free_meta_cache; + err = f2fs_get_valid_checkpoint(sbi); if (err) { f2fs_err(sbi, "Failed to get valid F2FS checkpoint"); - goto free_meta_cache; + goto free_node_cache; } if (__is_set_ckpt_flags(F2FS_CKPT(sbi), CP_QUOTA_NEED_FSCK_FLAG)) @@ -5546,6 +5551,8 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) free_devices: destroy_device_list(sbi); kvfree(sbi->ckpt); +free_node_cache: + f2fs_destroy_cache(NODE_CACHE(sbi)); free_meta_cache: f2fs_destroy_cache(META_CACHE(sbi)); free_page_array_cache: -- 2.49.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [PATCH v1 07/12] f2fs: cache: use node cache 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel -1 siblings, 0 replies; 34+ messages in thread From: Chao Yu @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-f2fs-devel, linux-kernel, Chao Yu This patch migrates F2FS node block caching from the fake VFS inode page cache (sbi->node_inode) to node cache (sbi->node_blocks). It updates node related helpers to use node cache APIs and reference to struct f2fs_cached_block, and removes sbi->node_inode, especially, unifies inline and regular dentry block handling via struct f2fs_dentry_block_ref. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/acl.c | 26 +- fs/f2fs/acl.h | 8 +- fs/f2fs/cache.c | 7 +- fs/f2fs/cache.h | 1 + fs/f2fs/checkpoint.c | 7 +- fs/f2fs/compress.c | 18 +- fs/f2fs/data.c | 236 +++++------ fs/f2fs/debug.c | 12 +- fs/f2fs/dir.c | 168 ++++---- fs/f2fs/extent_cache.c | 14 +- fs/f2fs/f2fs.h | 223 +++++----- fs/f2fs/file.c | 74 ++-- fs/f2fs/gc.c | 44 +- fs/f2fs/inline.c | 279 ++++++------- fs/f2fs/inode.c | 155 ++++--- fs/f2fs/namei.c | 118 +++--- fs/f2fs/node.c | 875 +++++++++++++++++++--------------------- fs/f2fs/node.h | 107 +++-- fs/f2fs/recovery.c | 115 +++--- fs/f2fs/segment.c | 59 +-- fs/f2fs/segment.h | 20 - fs/f2fs/shrinker.c | 3 +- fs/f2fs/super.c | 56 +-- fs/f2fs/xattr.c | 123 +++--- fs/f2fs/xattr.h | 12 +- include/linux/f2fs_fs.h | 1 - 26 files changed, 1329 insertions(+), 1432 deletions(-) diff --git a/fs/f2fs/acl.c b/fs/f2fs/acl.c index d3253549173e..34c9aa279040 100644 --- a/fs/f2fs/acl.c +++ b/fs/f2fs/acl.c @@ -181,7 +181,7 @@ static void *f2fs_acl_to_disk(struct f2fs_sb_info *sbi, } static struct posix_acl *__f2fs_get_acl(struct inode *inode, int type, - struct folio *dfolio) + struct f2fs_cached_block *entry) { int name_index = F2FS_XATTR_INDEX_POSIX_ACL_DEFAULT; void *value = NULL; @@ -191,13 +191,13 @@ static struct posix_acl *__f2fs_get_acl(struct inode *inode, int type, if (type == ACL_TYPE_ACCESS) name_index = F2FS_XATTR_INDEX_POSIX_ACL_ACCESS; - retval = f2fs_getxattr(inode, name_index, "", NULL, 0, dfolio); + retval = f2fs_getxattr(inode, name_index, "", NULL, 0, entry); if (retval > 0) { value = f2fs_kmalloc(F2FS_I_SB(inode), retval, GFP_F2FS_ZERO); if (!value) return ERR_PTR(-ENOMEM); retval = f2fs_getxattr(inode, name_index, "", value, - retval, dfolio); + retval, entry); } if (retval > 0) @@ -242,7 +242,7 @@ static int f2fs_acl_update_mode(struct mnt_idmap *idmap, static int __f2fs_set_acl(struct mnt_idmap *idmap, struct inode *inode, int type, - struct posix_acl *acl, struct folio *ifolio) + struct posix_acl *acl, struct f2fs_cached_block *ientry) { int name_index; void *value = NULL; @@ -253,7 +253,7 @@ static int __f2fs_set_acl(struct mnt_idmap *idmap, switch (type) { case ACL_TYPE_ACCESS: name_index = F2FS_XATTR_INDEX_POSIX_ACL_ACCESS; - if (acl && !ifolio) { + if (acl && !ientry) { error = f2fs_acl_update_mode(idmap, inode, &mode, &acl); if (error) return error; @@ -279,7 +279,7 @@ static int __f2fs_set_acl(struct mnt_idmap *idmap, } } - error = f2fs_setxattr(inode, name_index, "", value, size, ifolio, 0); + error = f2fs_setxattr(inode, name_index, "", value, size, ientry, 0); kfree(value); if (!error) @@ -374,7 +374,7 @@ static int f2fs_acl_create_masq(struct posix_acl *acl, umode_t *mode_p) static int f2fs_acl_create(struct inode *dir, umode_t *mode, struct posix_acl **default_acl, struct posix_acl **acl, - struct folio *dfolio) + struct f2fs_cached_block *entry) { struct posix_acl *p; struct posix_acl *clone; @@ -386,7 +386,7 @@ static int f2fs_acl_create(struct inode *dir, umode_t *mode, if (S_ISLNK(*mode) || !IS_POSIXACL(dir)) return 0; - p = __f2fs_get_acl(dir, ACL_TYPE_DEFAULT, dfolio); + p = __f2fs_get_acl(dir, ACL_TYPE_DEFAULT, entry); if (!p || p == ERR_PTR(-EOPNOTSUPP)) { *mode &= ~current_umask(); return 0; @@ -423,13 +423,13 @@ static int f2fs_acl_create(struct inode *dir, umode_t *mode, return ret; } -int f2fs_init_acl(struct inode *inode, struct inode *dir, struct folio *ifolio, - struct folio *dfolio) +int f2fs_init_acl(struct inode *inode, struct inode *dir, struct f2fs_cached_block *ientry, + struct f2fs_cached_block *dentry) { struct posix_acl *default_acl = NULL, *acl = NULL; int error; - error = f2fs_acl_create(dir, &inode->i_mode, &default_acl, &acl, dfolio); + error = f2fs_acl_create(dir, &inode->i_mode, &default_acl, &acl, dentry); if (error) return error; @@ -437,7 +437,7 @@ int f2fs_init_acl(struct inode *inode, struct inode *dir, struct folio *ifolio, if (default_acl) { error = __f2fs_set_acl(NULL, inode, ACL_TYPE_DEFAULT, - default_acl, ifolio); + default_acl, ientry); posix_acl_release(default_acl); } else { inode->i_default_acl = NULL; @@ -445,7 +445,7 @@ int f2fs_init_acl(struct inode *inode, struct inode *dir, struct folio *ifolio, if (acl) { if (!error) error = __f2fs_set_acl(NULL, inode, ACL_TYPE_ACCESS, - acl, ifolio); + acl, ientry); posix_acl_release(acl); } else { inode->i_acl = NULL; diff --git a/fs/f2fs/acl.h b/fs/f2fs/acl.h index 20e87e63c089..0f639367a0ab 100644 --- a/fs/f2fs/acl.h +++ b/fs/f2fs/acl.h @@ -36,14 +36,16 @@ struct f2fs_acl_header { struct posix_acl *f2fs_get_acl(struct inode *, int, bool); int f2fs_set_acl(struct mnt_idmap *, struct dentry *, struct posix_acl *, int); -int f2fs_init_acl(struct inode *, struct inode *, struct folio *ifolio, - struct folio *dfolio); +int f2fs_init_acl(struct inode *inode, struct inode *dir, + struct f2fs_cached_block *ientry, + struct f2fs_cached_block *dentry); #else #define f2fs_get_acl NULL #define f2fs_set_acl NULL static inline int f2fs_init_acl(struct inode *inode, struct inode *dir, - struct folio *ifolio, struct folio *dfolio) + struct f2fs_cached_block *ientry, + struct f2fs_cached_block *dentry) { return 0; } diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index 6c9b7a6d2313..3176d62ce25b 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -29,7 +29,7 @@ void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, return; /* submit cached bio */ - f2fs_submit_merged_write_cache(entry, type); + f2fs_submit_merged_write_cache(entry->cache->sbi, entry, 0, type); wait_on_bit_io(&entry->state, F2FS_BLOCK_WRITEBACK, TASK_UNINTERRUPTIBLE); @@ -67,8 +67,8 @@ bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry) f2fs_cache_set_uptodate(entry); #ifdef CONFIG_F2FS_CHECK_FS - if (cache->type == F2FS_NODE_CACHE && IS_INODE(cache_folio(entry))) - f2fs_inode_chksum_set(cache->sbi, cache_folio(entry)); + if (f2fs_is_node_cache(entry) && IS_INODE(entry)) + f2fs_inode_chksum_set(cache->sbi, entry); #endif if (f2fs_cache_test_dirty(entry)) @@ -640,6 +640,7 @@ static int f2fs_cache_writeback_kthread(void *data) continue; f2fs_write_meta_caches(sbi); + f2fs_write_node_caches(sbi); } return 0; } diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index d843b9caba47..397019cfb861 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -52,6 +52,7 @@ struct f2fs_cached_block_list { }; #define IS_META_CACHE(cache) (cache->type == F2FS_META_CACHE) +#define IS_NODE_CACHE(cache) (cache->type == F2FS_NODE_CACHE) /* Flags for f2fs_cached_block state */ enum f2fs_cached_state { diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c index 1a7083540b82..508132652693 100644 --- a/fs/f2fs/checkpoint.c +++ b/fs/f2fs/checkpoint.c @@ -1489,10 +1489,6 @@ static bool __need_flush_quota(struct f2fs_sb_info *sbi) */ static int block_operations(struct f2fs_sb_info *sbi) { - struct writeback_control wbc = { - .sync_mode = WB_SYNC_ALL, - .nr_to_write = LONG_MAX, - }; int err = 0, cnt = 0; /* @@ -1556,7 +1552,8 @@ static int block_operations(struct f2fs_sb_info *sbi) if (get_pages(sbi, F2FS_DIRTY_NODES)) { f2fs_up_write(&sbi->node_write); atomic_inc(&sbi->wb_sync_req[NODE]); - err = f2fs_sync_node_pages(sbi, &wbc, false, FS_CP_NODE_IO); + err = f2fs_writeback_node_caches(sbi, LONG_MAX, + true, false, FS_CP_NODE_IO); atomic_dec(&sbi->wb_sync_req[NODE]); if (err) { f2fs_up_write(&sbi->node_change); diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c index bb749f6257a1..676a5559357f 100644 --- a/fs/f2fs/compress.c +++ b/fs/f2fs/compress.c @@ -909,7 +909,7 @@ bool f2fs_sanity_check_cluster(struct dnode_of_data *dn) } for (i = 1, count = 1; i < cluster_size; i++, count++) { - block_t blkaddr = data_blkaddr(dn->inode, dn->node_folio, + block_t blkaddr = data_blkaddr(dn->inode, dn->node_entry, dn->ofs_in_node + i); /* [COMPR_ADDR, ..., COMPR_ADDR] */ @@ -950,7 +950,7 @@ static int __f2fs_get_cluster_blocks(struct inode *inode, int count, i; for (i = 0, count = 0; i < cluster_size; i++) { - block_t blkaddr = data_blkaddr(dn->inode, dn->node_folio, + block_t blkaddr = data_blkaddr(dn->inode, dn->node_entry, dn->ofs_in_node + i); if (__is_valid_data_blkaddr(blkaddr)) @@ -1146,7 +1146,7 @@ static int prepare_compress_overwrite(struct compress_ctx *cc, goto release_and_retry; } - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); f2fs_compress_ctx_add_page(cc, folio); if (!folio_test_uptodate(folio)) { @@ -1324,7 +1324,7 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc, goto out_unlock_op; for (i = 0; i < cc->cluster_size; i++) { - if (data_blkaddr(dn.inode, dn.node_folio, + if (data_blkaddr(dn.inode, dn.node_entry, dn.ofs_in_node + i) == NULL_ADDR) goto out_put_dnode; } @@ -1356,7 +1356,7 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc, page_folio(cc->rpages[i + 1])->index, cic); fio.compressed_page = cc->cpages[i]; - fio.old_blkaddr = data_blkaddr(dn.inode, dn.node_folio, + fio.old_blkaddr = data_blkaddr(dn.inode, dn.node_entry, dn.ofs_in_node + i + 1); /* wait for GCed page writeback via generic cache */ @@ -1567,7 +1567,7 @@ static int f2fs_write_raw_pages(struct compress_ctx *cc, if (folio_test_writeback(folio)) { if (wbc->sync_mode == WB_SYNC_NONE) goto continue_unlock; - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); } if (!folio_clear_dirty_for_io(folio)) @@ -1899,14 +1899,14 @@ void f2fs_put_folio_dic(struct folio *folio, bool in_task) unsigned int f2fs_cluster_blocks_are_contiguous(struct dnode_of_data *dn, unsigned int ofs_in_node) { - bool compressed = data_blkaddr(dn->inode, dn->node_folio, + bool compressed = data_blkaddr(dn->inode, dn->node_entry, ofs_in_node) == COMPRESS_ADDR; int i = compressed ? 1 : 0; - block_t first_blkaddr = data_blkaddr(dn->inode, dn->node_folio, + block_t first_blkaddr = data_blkaddr(dn->inode, dn->node_entry, ofs_in_node + i); for (i += 1; i < F2FS_I(dn->inode)->i_cluster_size; i++) { - block_t blkaddr = data_blkaddr(dn->inode, dn->node_folio, + block_t blkaddr = data_blkaddr(dn->inode, dn->node_entry, ofs_in_node + i); if (!__is_valid_data_blkaddr(blkaddr)) diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index 92c3293f0a1e..af05a1f6a00f 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -58,15 +58,13 @@ bool f2fs_is_cp_guaranteed(const struct folio *folio) { struct address_space *mapping = folio->mapping; struct inode *inode; - struct f2fs_sb_info *sbi; if (fscrypt_is_bounce_folio(folio)) return folio_test_f2fs_gcing(fscrypt_pagecache_folio(folio)); inode = mapping->host; - if (inode->i_ino == F2FS_NODE_INO(sbi) || - S_ISDIR(inode->i_mode)) + if (S_ISDIR(inode->i_mode)) return true; if ((S_ISREG(inode->i_mode) && IS_NOQUOTA(inode)) || @@ -75,20 +73,6 @@ bool f2fs_is_cp_guaranteed(const struct folio *folio) return false; } -static enum count_type __read_io_type(struct folio *folio) -{ - struct address_space *mapping = folio->mapping; - - if (mapping) { - struct inode *inode = mapping->host; - struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - - if (inode->i_ino == F2FS_NODE_INO(sbi)) - return F2FS_RD_NODE; - } - return F2FS_RD_DATA; -} - /* postprocessing steps for read bios */ enum bio_post_read_step { #ifdef CONFIG_FS_ENCRYPTION @@ -167,13 +151,7 @@ static void f2fs_finish_read_bio(struct bio *bio, bool in_task) } while (nr_pages--) - dec_page_count(F2FS_F_SB(folio), __read_io_type(folio)); - - if (bio->bi_status == BLK_STS_OK && - F2FS_F_SB(folio)->node_inode && is_node_folio(folio) && - f2fs_sanity_check_node_footer(F2FS_F_SB(folio), - folio, folio->index, NODE_TYPE_REGULAR, true)) - bio->bi_status = BLK_STS_IOERR; + dec_page_count(F2FS_F_SB(folio), F2FS_RD_DATA); if (finished) folio_end_read(folio, bio->bi_status == BLK_STS_OK); @@ -376,14 +354,6 @@ static void f2fs_write_end_bio(struct bio *bio) } } - if (is_node_folio(folio)) { - f2fs_sanity_check_node_footer(sbi, folio, - folio->index, NODE_TYPE_REGULAR, true); - f2fs_bug_on(sbi, folio->index != nid_of_node(folio)); - } - if (f2fs_in_warm_node_list(folio)) - f2fs_del_fsync_node_entry(sbi, folio); - dec_page_count(sbi, type); /* @@ -444,6 +414,12 @@ static void f2fs_cache_read_end_io(struct bio *bio) next = entry->next_entry; entry->next_entry = NULL; + if (bio->bi_status == BLK_STS_OK && + f2fs_is_node_cache(entry) && + f2fs_sanity_check_node_footer(sbi, entry, + entry->index, NODE_TYPE_REGULAR, true)) + bio->bi_status = BLK_STS_IOERR; + if (bio->bi_status == BLK_STS_OK) f2fs_cache_set_uptodate(entry); @@ -474,6 +450,14 @@ static void f2fs_cache_write_end_io(struct bio *bio) next = entry->next_entry; entry->next_entry = NULL; + if (f2fs_is_node_cache(entry)) { + f2fs_sanity_check_node_footer(sbi, entry, + entry->index, NODE_TYPE_REGULAR, true); + f2fs_bug_on(sbi, entry->index != nid_of_node(entry)); + } + if (f2fs_in_warm_node_list(entry)) + f2fs_del_fsync_node_entry(sbi, entry); + dec_page_count(sbi, F2FS_WB_CP_DATA); if (!get_pages(sbi, F2FS_WB_CP_DATA) && @@ -681,14 +665,14 @@ static void __submit_merged_bio(struct f2fs_bio_info *io) } static bool __has_merged_page(struct bio *bio, struct inode *inode, - struct folio *folio, nid_t ino) + struct folio *folio) { struct folio_iter fi; if (!bio) return false; - if (!inode && !folio && !ino) + if (!inode && !folio) return true; if (f2fs_is_cache_bio(bio)) @@ -712,8 +696,6 @@ static bool __has_merged_page(struct bio *bio, struct inode *inode, return true; if (folio && folio == target) return true; - if (ino && ino == ino_of_node(target)) - return true; } return false; @@ -782,24 +764,23 @@ static void __f2fs_submit_merged_write(struct f2fs_sb_info *sbi, static void __submit_merged_write_cond(struct f2fs_sb_info *sbi, struct inode *inode, struct folio *folio, - nid_t ino, enum page_type type, bool writeback) + bool writeback) { enum temp_type temp; bool ret = true; - bool force = !inode && !folio && !ino; + bool force = !inode && !folio; for (temp = HOT; temp < NR_TEMP_TYPE; temp++) { - if (!force) { - enum page_type btype = PAGE_TYPE_OF_BIO(type); - struct f2fs_bio_info *io = sbi->write_io[btype] + temp; + if (!force) { + struct f2fs_bio_info *io = sbi->write_io[DATA] + temp; struct f2fs_lock_context lc; f2fs_down_read_trace(&io->io_rwsem, &lc); - ret = __has_merged_page(io->bio, inode, folio, ino); + ret = __has_merged_page(io->bio, inode, folio); f2fs_up_read_trace(&io->io_rwsem, &lc); } if (ret) { - __f2fs_submit_merged_write(sbi, type, temp); + __f2fs_submit_merged_write(sbi, DATA, temp); /* * For waitting writebck case, if the bio owned by the * folio is already submitted, we do not need to submit @@ -808,33 +789,23 @@ static void __submit_merged_write_cond(struct f2fs_sb_info *sbi, if (writeback) break; } - - /* TODO: use HOT temp only for meta pages now. */ - if (type >= META) - break; } } -void f2fs_submit_merged_write(struct f2fs_sb_info *sbi, enum page_type type) -{ - __submit_merged_write_cond(sbi, NULL, NULL, 0, type, false); -} - void f2fs_submit_merged_write_cond(struct f2fs_sb_info *sbi, - struct inode *inode, struct folio *folio, - nid_t ino, enum page_type type) + struct inode *inode, struct folio *folio) { - __submit_merged_write_cond(sbi, inode, folio, ino, type, false); + __submit_merged_write_cond(sbi, inode, folio, false); } void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, - struct folio *folio, enum page_type type) + struct folio *folio) { - __submit_merged_write_cond(sbi, NULL, folio, 0, type, true); + __submit_merged_write_cond(sbi, NULL, folio, true); } static bool __has_merged_cache(struct bio *bio, - struct f2fs_cached_block *target) + struct f2fs_cached_block *target, nid_t ino) { struct f2fs_cached_block *entry; @@ -846,27 +817,32 @@ static bool __has_merged_cache(struct bio *bio, while (entry) { if (target && entry == target) return true; + if (ino && ino_of_node(entry) == ino) + return true; entry = entry->next_entry; } return false; } -bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, - enum page_type type) +bool f2fs_submit_merged_write_cache(struct f2fs_sb_info *sbi, + struct f2fs_cached_block *entry, + nid_t ino, enum page_type type) { - struct f2fs_sb_info *sbi = entry->cache->sbi; enum temp_type temp; bool ret = false; + bool force = !entry && !ino; for (temp = HOT; temp < NR_TEMP_TYPE; temp++) { enum page_type btype = PAGE_TYPE_OF_BIO(type); struct f2fs_bio_info *io = sbi->write_io[btype] + temp; struct f2fs_lock_context lc; - bool merged; + bool merged = true; - f2fs_down_read_trace(&io->io_rwsem, &lc); - merged = __has_merged_cache(io->bio, entry); - f2fs_up_read_trace(&io->io_rwsem, &lc); + if (!force) { + f2fs_down_read_trace(&io->io_rwsem, &lc); + merged = __has_merged_cache(io->bio, entry, ino); + f2fs_up_read_trace(&io->io_rwsem, &lc); + } if (merged) { __f2fs_submit_merged_write(sbi, type, temp); @@ -880,6 +856,14 @@ bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, return ret; } +void f2fs_submit_merged_write(struct f2fs_sb_info *sbi, enum page_type type) +{ + if (type == DATA) + __submit_merged_write_cond(sbi, NULL, NULL, false); + else + f2fs_submit_merged_write_cache(sbi, NULL, 0, type); +} + void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi) { f2fs_submit_merged_write(sbi, DATA); @@ -916,7 +900,7 @@ int f2fs_submit_page_bio(struct f2fs_io_info *fio) wbc_account_cgroup_owner(fio->io_wbc, fio_folio, PAGE_SIZE); inc_page_count(fio->sbi, is_read_io(fio->op) ? - __read_io_type(data_folio) : WB_DATA_TYPE(fio->folio, false)); + F2FS_RD_DATA : WB_DATA_TYPE(fio->folio, false)); if (is_read_io(bio_op(bio))) f2fs_submit_read_bio(fio->sbi, bio, fio->type); @@ -1052,8 +1036,8 @@ void f2fs_submit_merged_ipu_write(struct f2fs_sb_info *sbi, if (target) found = (target == be->bio); else - found = __has_merged_page(be->bio, NULL, - folio, 0); + found = __has_merged_page(be->bio, + NULL, folio); if (found) break; } @@ -1069,8 +1053,8 @@ void f2fs_submit_merged_ipu_write(struct f2fs_sb_info *sbi, if (target) found = (target == be->bio); else - found = __has_merged_page(be->bio, NULL, - folio, 0); + found = __has_merged_page(be->bio, + NULL, folio); if (found) { target = be->bio; del_bio_entry(be); @@ -1461,7 +1445,7 @@ static void f2fs_submit_page_read(struct inode *inode, struct fsverity_info *vi, static void __set_data_blkaddr(struct dnode_of_data *dn, block_t blkaddr) { - __le32 *addr = get_dnode_addr(dn->inode, dn->node_folio); + __le32 *addr = get_dnode_addr(dn->inode, dn->node_entry); dn->data_blkaddr = blkaddr; addr[dn->ofs_in_node] = cpu_to_le32(dn->data_blkaddr); @@ -1475,9 +1459,9 @@ static void __set_data_blkaddr(struct dnode_of_data *dn, block_t blkaddr) */ void f2fs_set_data_blkaddr(struct dnode_of_data *dn, block_t blkaddr) { - f2fs_folio_wait_writeback(dn->node_folio, NODE, true, true); + f2fs_cache_wait_writeback(dn->node_entry); __set_data_blkaddr(dn, blkaddr); - if (folio_mark_dirty(dn->node_folio)) + if (f2fs_mark_cache_dirty(dn->node_entry)) dn->node_changed = true; } @@ -1505,7 +1489,7 @@ int f2fs_reserve_new_blocks(struct dnode_of_data *dn, blkcnt_t count) trace_f2fs_reserve_new_blocks(dn->inode, dn->nid, dn->ofs_in_node, count); - f2fs_folio_wait_writeback(dn->node_folio, NODE, true, true); + f2fs_cache_wait_writeback(dn->node_entry); for (; count > 0; dn->ofs_in_node++) { block_t blkaddr = f2fs_data_blkaddr(dn); @@ -1516,7 +1500,7 @@ int f2fs_reserve_new_blocks(struct dnode_of_data *dn, blkcnt_t count) } } - if (folio_mark_dirty(dn->node_folio)) + if (f2fs_mark_cache_dirty(dn->node_entry)) dn->node_changed = true; return 0; } @@ -1534,7 +1518,7 @@ int f2fs_reserve_new_block(struct dnode_of_data *dn) int f2fs_reserve_block(struct dnode_of_data *dn, pgoff_t index) { - bool need_put = dn->inode_folio ? false : true; + bool need_put = dn->inode_entry ? false : true; int err; err = f2fs_get_dnode_of_data(dn, index, ALLOC_NODE); @@ -1700,11 +1684,11 @@ struct folio *f2fs_get_lock_data_folio(struct inode *inode, pgoff_t index, * * Also, caller should grab and release a rwsem by calling f2fs_lock_op() and * f2fs_unlock_op(). - * Note that, ifolio is set only by make_empty_dir, and if any error occur, - * ifolio should be released by this function. + * Note that, ientry is set only by make_empty_dir, and if any error occur, + * ientry should be released by this function. */ struct folio *f2fs_get_new_data_folio(struct inode *inode, - struct folio *ifolio, pgoff_t index, bool new_i_size) + struct f2fs_cached_block *ientry, pgoff_t index, bool new_i_size) { struct address_space *mapping = inode->i_mapping; struct folio *folio; @@ -1714,20 +1698,20 @@ struct folio *f2fs_get_new_data_folio(struct inode *inode, folio = f2fs_grab_cache_folio(mapping, index, true); if (IS_ERR(folio)) { /* - * before exiting, we should make sure ifolio will be released + * before exiting, we should make sure ientry will be released * if any error occur. */ - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); return ERR_PTR(-ENOMEM); } - set_new_dnode(&dn, inode, ifolio, NULL, 0); + set_new_dnode(&dn, inode, ientry, NULL, 0); err = f2fs_reserve_block(&dn, index); if (err) { f2fs_folio_put(folio, true); return ERR_PTR(err); } - if (!ifolio) + if (!ientry) f2fs_put_dnode(&dn); if (folio_test_uptodate(folio)) @@ -1740,8 +1724,8 @@ struct folio *f2fs_get_new_data_folio(struct inode *inode, } else { f2fs_folio_put(folio, true); - /* if ifolio exists, blkaddr should be NEW_ADDR */ - f2fs_bug_on(F2FS_I_SB(inode), ifolio); + /* if ientry exists, blkaddr should be NEW_ADDR */ + f2fs_bug_on(F2FS_I_SB(inode), ientry); folio = f2fs_get_lock_data_folio(inode, index, true); if (IS_ERR(folio)) return folio; @@ -1778,7 +1762,7 @@ static int __allocate_data_block(struct dnode_of_data *dn, int seg_type) set_summary(&sum, dn->nid, dn->ofs_in_node, ni.version); old_blkaddr = dn->data_blkaddr; - err = f2fs_allocate_data_block(sbi, NULL, old_blkaddr, + err = f2fs_allocate_data_block(sbi, old_blkaddr, &dn->data_blkaddr, &sum, seg_type, NULL); if (err) { if (old_blkaddr == NULL_ADDR) @@ -1990,7 +1974,7 @@ int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map, int flag) start_pgofs = pgofs; prealloc = 0; last_ofs_in_node = ofs_in_node = dn.ofs_in_node; - end_offset = ADDRS_PER_PAGE(dn.node_folio, inode); + end_offset = ADDRS_PER_PAGE(dn.node_entry, inode); next_block: blkaddr = f2fs_data_blkaddr(&dn); @@ -2241,15 +2225,15 @@ static int f2fs_xattr_fiemap(struct inode *inode, if (f2fs_has_inline_xattr(inode)) { int offset; - struct folio *folio = f2fs_grab_cache_folio(NODE_MAPPING(sbi), - inode->i_ino, false); + struct f2fs_cached_block *entry = + f2fs_grab_node_cache(sbi, inode->i_ino); - if (IS_ERR(folio)) - return PTR_ERR(folio); + if (IS_ERR(entry)) + return PTR_ERR(entry); err = f2fs_get_node_info(sbi, inode->i_ino, &ni, false); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); return err; } @@ -2261,7 +2245,7 @@ static int f2fs_xattr_fiemap(struct inode *inode, phys += offset; len = inline_xattr_size(inode); - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); flags = FIEMAP_EXTENT_DATA_INLINE | FIEMAP_EXTENT_NOT_ALIGNED; @@ -2275,22 +2259,22 @@ static int f2fs_xattr_fiemap(struct inode *inode, } if (xnid) { - struct folio *folio = f2fs_grab_cache_folio(NODE_MAPPING(sbi), - xnid, false); + struct f2fs_cached_block *entry = + f2fs_grab_node_cache(sbi, xnid); - if (IS_ERR(folio)) - return PTR_ERR(folio); + if (IS_ERR(entry)) + return PTR_ERR(entry); err = f2fs_get_node_info(sbi, xnid, &ni, false); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); return err; } phys = F2FS_BLK_TO_BYTES(ni.blk_addr); len = inode->i_sb->s_blocksize; - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); flags = FIEMAP_EXTENT_LAST; } @@ -2649,7 +2633,7 @@ int f2fs_read_multi_pages(struct compress_ctx *cc, struct bio **bio_ret, for (i = 1; i < cc->cluster_size; i++) { block_t blkaddr; - blkaddr = from_dnode ? data_blkaddr(dn.inode, dn.node_folio, + blkaddr = from_dnode ? data_blkaddr(dn.inode, dn.node_entry, dn.ofs_in_node + i) : ei.blk + i - 1; @@ -2683,7 +2667,7 @@ int f2fs_read_multi_pages(struct compress_ctx *cc, struct bio **bio_ret, block_t blkaddr; struct bio_post_read_ctx *ctx; - blkaddr = from_dnode ? data_blkaddr(dn.inode, dn.node_folio, + blkaddr = from_dnode ? data_blkaddr(dn.inode, dn.node_entry, dn.ofs_in_node + i + 1) : ei.blk + i; @@ -3714,7 +3698,7 @@ static int f2fs_write_cache_pages(struct address_space *mapping, if (folio_test_writeback(folio)) { if (wbc->sync_mode == WB_SYNC_NONE) goto continue_unlock; - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); } if (!folio_clear_dirty_for_io(folio)) @@ -3796,8 +3780,8 @@ static int f2fs_write_cache_pages(struct address_space *mapping, mapping->writeback_index = done_index; if (nwritten) - f2fs_submit_merged_write_cond(F2FS_M_SB(mapping), mapping->host, - NULL, 0, DATA); + f2fs_submit_merged_write_cond(F2FS_M_SB(mapping), + mapping->host, NULL); /* submit cached bio of IPU write */ if (bio) f2fs_submit_merged_ipu_write(sbi, &bio, NULL); @@ -3975,7 +3959,7 @@ static int prepare_write_begin(struct f2fs_sb_info *sbi, pgoff_t index = folio->index; struct dnode_of_data dn; struct f2fs_lock_context lc; - struct folio *ifolio; + struct f2fs_cached_block *ientry; bool locked = false; int flag = F2FS_GET_BLOCK_PRE_AIO; int err = 0; @@ -4005,20 +3989,20 @@ static int prepare_write_begin(struct f2fs_sb_info *sbi, restart: /* check inline_data */ - ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(ifolio)) { - err = PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(ientry)) { + err = PTR_ERR(ientry); goto unlock_out; } - set_new_dnode(&dn, inode, ifolio, ifolio, 0); + set_new_dnode(&dn, inode, ientry, ientry, 0); if (f2fs_has_inline_data(inode)) { if (pos + len <= MAX_INLINE_DATA(inode)) { - f2fs_do_read_inline_data(folio, ifolio); + f2fs_do_read_inline_data(folio, ientry); set_inode_flag(inode, FI_DATA_EXIST); if (inode->i_nlink) - folio_set_f2fs_inline(ifolio); + f2fs_cache_set_inline(ientry); goto out; } err = f2fs_convert_inline_folio(&dn, folio); @@ -4065,14 +4049,14 @@ static int __find_data_block(struct inode *inode, pgoff_t index, block_t *blk_addr) { struct dnode_of_data dn; - struct folio *ifolio; + struct f2fs_cached_block *ientry; int err = 0; - ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); - set_new_dnode(&dn, inode, ifolio, ifolio, 0); + set_new_dnode(&dn, inode, ientry, ientry, 0); if (!f2fs_lookup_read_extent_cache_block(inode, index, &dn.data_blkaddr)) { @@ -4094,17 +4078,17 @@ static int __reserve_data_block(struct inode *inode, pgoff_t index, struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct dnode_of_data dn; struct f2fs_lock_context lc; - struct folio *ifolio; + struct f2fs_cached_block *ientry; int err = 0; f2fs_map_lock(sbi, &lc, F2FS_GET_BLOCK_PRE_AIO); - ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(ifolio)) { - err = PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(ientry)) { + err = PTR_ERR(ientry); goto unlock_out; } - set_new_dnode(&dn, inode, ifolio, ifolio, 0); + set_new_dnode(&dn, inode, ientry, ientry, 0); if (!f2fs_lookup_read_extent_cache_block(dn.inode, index, &dn.data_blkaddr)) @@ -4260,7 +4244,7 @@ static int f2fs_write_begin(const struct kiocb *iocb, } } - f2fs_folio_wait_writeback(folio, DATA, false, true); + f2fs_folio_wait_writeback(folio, false, true); if (len == folio_size(folio) || folio_test_uptodate(folio)) return 0; @@ -4377,12 +4361,8 @@ void f2fs_invalidate_folio(struct folio *folio, size_t offset, size_t length) return; if (folio_test_dirty(folio)) { - if (inode->i_ino == F2FS_NODE_INO(sbi)) { - dec_page_count(sbi, F2FS_DIRTY_NODES); - } else { - inode_dec_dirty_pages(inode); - f2fs_remove_dirty_inode(inode); - } + inode_dec_dirty_pages(inode); + f2fs_remove_dirty_inode(inode); } if (offset || length != folio_size(folio)) diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c index 14059a50739c..bedaade92677 100644 --- a/fs/f2fs/debug.c +++ b/fs/f2fs/debug.c @@ -222,8 +222,7 @@ static void update_general_status(struct f2fs_sb_info *sbi) si->free_secs = free_sections(sbi); si->prefree_count = prefree_segments(sbi); si->dirty_count = dirty_segments(sbi); - if (sbi->node_inode) - si->node_pages = NODE_MAPPING(sbi)->nrpages; + si->node_caches = NODE_CACHE(sbi)->num_entries; si->meta_caches = META_CACHE(sbi)->num_entries; #ifdef CONFIG_F2FS_FS_COMPRESSION if (sbi->compress_inode) { @@ -382,13 +381,10 @@ static void update_mem_info(struct f2fs_sb_info *sbi) } si->page_mem = 0; - if (sbi->node_inode) { - unsigned long npages = NODE_MAPPING(sbi)->nrpages; - - si->page_mem += (unsigned long long)npages << PAGE_SHIFT; - } si->page_mem += (unsigned long long)META_CACHE(sbi)->num_entries << PAGE_SHIFT; si->cache_mem += META_CACHE(sbi)->num_entries * sizeof(struct f2fs_cached_block); + si->page_mem += (unsigned long long)NODE_CACHE(sbi)->num_entries << PAGE_SHIFT; + si->cache_mem += NODE_CACHE(sbi)->num_entries * sizeof(struct f2fs_cached_block); #ifdef CONFIG_F2FS_FS_COMPRESSION if (sbi->compress_inode) { unsigned long npages = COMPRESS_MAPPING(sbi)->nrpages; @@ -696,7 +692,7 @@ static int stat_show(struct seq_file *s, void *v) si->aw_cnt, si->max_aw_cnt); seq_printf(s, " - compress: %4d, hit:%8d\n", si->compress_pages, si->compress_page_hit); seq_printf(s, " - nodes: %4d in %4d\n", - si->ndirty_node, si->node_pages); + si->ndirty_node, si->node_caches); seq_printf(s, " - dents: %4d in dirs:%4d (%4d)\n", si->ndirty_dent, si->ndirty_dirs, si->ndirty_all); seq_printf(s, " - data: %4d in files:%4d\n", diff --git a/fs/f2fs/dir.c b/fs/f2fs/dir.c index fd0e2cd31a81..20be3a28509b 100644 --- a/fs/f2fs/dir.c +++ b/fs/f2fs/dir.c @@ -282,7 +282,7 @@ struct f2fs_dir_entry *f2fs_find_target_dentry(const struct f2fs_dentry_ptr *d, static struct f2fs_dir_entry *find_in_level(struct inode *dir, unsigned int level, const struct f2fs_filename *fname, - struct folio **res_folio, + void **dentry_block, bool use_hash) { int s = GET_DENTRY_SLOTS(fname->disk_name.len); @@ -313,7 +313,7 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir, bidx = next_pgofs; continue; } else { - *res_folio = dentry_folio; + *dentry_block = dentry_folio; break; } } @@ -321,11 +321,11 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir, de = find_in_block(dir, dentry_folio, fname, &max_slots, use_hash); if (IS_ERR(de)) { f2fs_folio_put(dentry_folio, false); - *res_folio = ERR_CAST(de); + *dentry_block = ERR_CAST(de); de = NULL; break; } else if (de) { - *res_folio = dentry_folio; + *dentry_block = dentry_folio; break; } @@ -352,7 +352,7 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir, struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir, const struct f2fs_filename *fname, - struct folio **res_folio) + void **dentry_block) { unsigned long npages = dir_blocks(dir); struct f2fs_dir_entry *de = NULL; @@ -360,13 +360,13 @@ struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir, unsigned int level; bool use_hash = true; - *res_folio = NULL; + *dentry_block = NULL; #if IS_ENABLED(CONFIG_UNICODE) start_find_entry: #endif if (f2fs_has_inline_dentry(dir)) { - de = f2fs_find_in_inline_dir(dir, fname, res_folio, use_hash); + de = f2fs_find_in_inline_dir(dir, fname, dentry_block, use_hash); goto out; } @@ -382,8 +382,8 @@ struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir, } for (level = 0; level < max_depth; level++) { - de = find_in_level(dir, level, fname, res_folio, use_hash); - if (de || IS_ERR(*res_folio)) + de = find_in_level(dir, level, fname, dentry_block, use_hash); + if (de || IS_ERR(*dentry_block)) break; } @@ -408,7 +408,7 @@ struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir, * Entry is guaranteed to be valid. */ struct f2fs_dir_entry *f2fs_find_entry(struct inode *dir, - const struct qstr *child, struct folio **res_folio) + const struct qstr *child, void **dentry_block) { struct f2fs_dir_entry *de = NULL; struct f2fs_filename fname; @@ -417,67 +417,78 @@ struct f2fs_dir_entry *f2fs_find_entry(struct inode *dir, err = f2fs_setup_filename(dir, child, 1, &fname); if (err) { if (err == -ENOENT) - *res_folio = NULL; + *dentry_block = NULL; else - *res_folio = ERR_PTR(err); + *dentry_block = ERR_PTR(err); return NULL; } - de = __f2fs_find_entry(dir, &fname, res_folio); + de = __f2fs_find_entry(dir, &fname, dentry_block); f2fs_free_filename(&fname); return de; } -struct f2fs_dir_entry *f2fs_parent_dir(struct inode *dir, struct folio **f) +struct f2fs_dir_entry *f2fs_parent_dir(struct inode *dir, void **dentry_block) { - return f2fs_find_entry(dir, &dotdot_name, f); + return f2fs_find_entry(dir, &dotdot_name, dentry_block); } ino_t f2fs_inode_by_name(struct inode *dir, const struct qstr *qstr, - struct folio **folio) + void **dentry_block) { ino_t res = 0; struct f2fs_dir_entry *de; - de = f2fs_find_entry(dir, qstr, folio); + de = f2fs_find_entry(dir, qstr, dentry_block); if (de) { res = le32_to_cpu(de->ino); - f2fs_folio_put(*folio, false); + f2fs_put_dentry_block(*dentry_block, false); } return res; } void f2fs_set_link(struct inode *dir, struct f2fs_dir_entry *de, - struct folio *folio, struct inode *inode) + void *dentry_block, struct inode *inode) { - enum page_type type = f2fs_has_inline_dentry(dir) ? NODE : DATA; + if (f2fs_dentry_is_cache(dentry_block)) { + struct f2fs_cached_block *entry = + f2fs_dentry_cache(dentry_block); + + f2fs_lock_cache(entry); + f2fs_cache_wait_writeback(entry); + de->ino = cpu_to_le32(inode->i_ino); + de->file_type = fs_umode_to_ftype(inode->i_mode); + f2fs_mark_cache_dirty(entry); + } else { + struct folio *folio = f2fs_dentry_folio(dentry_block); - folio_lock(folio); - f2fs_folio_wait_writeback(folio, type, true, true); - de->ino = cpu_to_le32(inode->i_ino); - de->file_type = fs_umode_to_ftype(inode->i_mode); - folio_mark_dirty(folio); + folio_lock(folio); + f2fs_folio_wait_writeback(folio, true, true); + de->ino = cpu_to_le32(inode->i_ino); + de->file_type = fs_umode_to_ftype(inode->i_mode); + folio_mark_dirty(folio); + } inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir)); f2fs_mark_inode_dirty_sync(dir, true); - f2fs_folio_put(folio, true); + f2fs_put_dentry_block(dentry_block, true); } static void init_dent_inode(struct inode *dir, struct inode *inode, const struct f2fs_filename *fname, - struct folio *ifolio) + struct f2fs_cached_block *ientry) { struct f2fs_inode *ri; if (!fname) /* tmpfile case? */ return; - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_cache_wait_writeback(ientry); /* copy name info. to this inode folio */ - ri = F2FS_INODE(ifolio); + ri = F2FS_INODE(ientry); ri->i_namelen = cpu_to_le32(fname->disk_name.len); memcpy(ri->i_name, fname->disk_name.name, fname->disk_name.len); if (IS_ENCRYPTED(dir)) { @@ -498,7 +509,7 @@ static void init_dent_inode(struct inode *dir, struct inode *inode, file_lost_pino(inode); } } - folio_mark_dirty(ifolio); + f2fs_mark_cache_dirty(ientry); } void f2fs_do_make_empty_dir(struct inode *inode, struct inode *parent, @@ -515,16 +526,16 @@ void f2fs_do_make_empty_dir(struct inode *inode, struct inode *parent, } static int make_empty_dir(struct inode *inode, - struct inode *parent, struct folio *folio) + struct inode *parent, struct f2fs_cached_block *ientry) { struct folio *dentry_folio; struct f2fs_dentry_block *dentry_blk; struct f2fs_dentry_ptr d; if (f2fs_has_inline_dentry(inode)) - return f2fs_make_empty_inline_dir(inode, parent, folio); + return f2fs_make_empty_inline_dir(inode, parent, ientry); - dentry_folio = f2fs_get_new_data_folio(inode, folio, 0, true); + dentry_folio = f2fs_get_new_data_folio(inode, ientry, 0, true); if (IS_ERR(dentry_folio)) return PTR_ERR(dentry_folio); @@ -538,50 +549,50 @@ static int make_empty_dir(struct inode *inode, return 0; } -struct folio *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir, - const struct f2fs_filename *fname, struct folio *dfolio) +struct f2fs_cached_block *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir, + const struct f2fs_filename *fname, struct f2fs_cached_block *dentry) { - struct folio *folio; + struct f2fs_cached_block *ientry; int err; if (is_inode_flag_set(inode, FI_NEW_INODE)) { - folio = f2fs_new_inode_folio(inode); - if (IS_ERR(folio)) - return folio; + ientry = f2fs_new_inode_cache(inode); + if (IS_ERR(ientry)) + return ientry; if (S_ISDIR(inode->i_mode)) { /* in order to handle error case */ - folio_get(folio); - err = make_empty_dir(inode, dir, folio); + f2fs_cache_get(ientry); + err = make_empty_dir(inode, dir, ientry); if (err) { - folio_lock(folio); + f2fs_lock_cache(ientry); goto put_error; } - folio_put(folio); + f2fs_put_cache(ientry, false); } - err = f2fs_init_acl(inode, dir, folio, dfolio); + err = f2fs_init_acl(inode, dir, ientry, dentry); if (err) goto put_error; err = f2fs_init_security(inode, dir, fname ? fname->usr_fname : NULL, - folio); + ientry); if (err) goto put_error; if (IS_ENCRYPTED(inode)) { - err = fscrypt_set_context(inode, folio); + err = fscrypt_set_context(inode, ientry); if (err) goto put_error; } } else { - folio = f2fs_get_inode_folio(F2FS_I_SB(dir), inode->i_ino); - if (IS_ERR(folio)) - return folio; + ientry = f2fs_get_inode_cache(F2FS_I_SB(dir), inode->i_ino); + if (IS_ERR(ientry)) + return ientry; } - init_dent_inode(dir, inode, fname, folio); + init_dent_inode(dir, inode, fname, ientry); /* * This file should be checkpointed during fsync. @@ -598,12 +609,12 @@ struct folio *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir, f2fs_remove_orphan_inode(F2FS_I_SB(dir), inode->i_ino); f2fs_i_links_write(inode, true); } - return folio; + return ientry; put_error: clear_nlink(inode); - f2fs_update_inode(inode, folio); - f2fs_folio_put(folio, true); + f2fs_update_inode(inode, ientry); + f2fs_put_cache(ientry, true); return ERR_PTR(err); } @@ -645,14 +656,14 @@ int f2fs_room_for_filename(const void *bitmap, int slots, int max_slots) goto next; } -bool f2fs_has_enough_room(struct inode *dir, struct folio *ifolio, +bool f2fs_has_enough_room(struct inode *dir, struct f2fs_cached_block *ientry, const struct f2fs_filename *fname) { struct f2fs_dentry_ptr d; unsigned int bit_pos; int slots = GET_DENTRY_SLOTS(fname->disk_name.len); - make_dentry_ptr_inline(dir, &d, inline_data_addr(dir, ifolio)); + make_dentry_ptr_inline(dir, &d, inline_data_addr(dir, ientry)); bit_pos = f2fs_room_for_filename(d.bitmap, slots, d.max); @@ -692,7 +703,7 @@ int f2fs_add_regular_entry(struct inode *dir, const struct f2fs_filename *fname, struct folio *dentry_folio = NULL; struct f2fs_dentry_block *dentry_blk = NULL; struct f2fs_dentry_ptr d; - struct folio *folio = NULL; + struct f2fs_cached_block *entry = NULL; int slots, err = 0; level = 0; @@ -739,13 +750,13 @@ int f2fs_add_regular_entry(struct inode *dir, const struct f2fs_filename *fname, ++level; goto start; add_dentry: - f2fs_folio_wait_writeback(dentry_folio, DATA, true, true); + f2fs_folio_wait_writeback(dentry_folio, true, true); if (inode) { f2fs_down_write(&F2FS_I(inode)->i_sem); - folio = f2fs_init_inode_metadata(inode, dir, fname, NULL); - if (IS_ERR(folio)) { - err = PTR_ERR(folio); + entry = f2fs_init_inode_metadata(inode, dir, fname, NULL); + if (IS_ERR(entry)) { + err = PTR_ERR(entry); goto fail; } } @@ -761,9 +772,9 @@ int f2fs_add_regular_entry(struct inode *dir, const struct f2fs_filename *fname, /* synchronize inode page's data from inode cache */ if (is_inode_flag_set(inode, FI_NEW_INODE)) - f2fs_update_inode(inode, folio); + f2fs_update_inode(inode, entry); - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); } f2fs_update_parent_metadata(dir, inode, current_depth); @@ -805,7 +816,7 @@ int f2fs_do_add_link(struct inode *dir, const struct qstr *name, struct inode *inode, nid_t ino, umode_t mode) { struct f2fs_filename fname; - struct folio *folio = NULL; + void *dentry_blk = NULL; struct f2fs_dir_entry *de = NULL; int err; @@ -821,14 +832,14 @@ int f2fs_do_add_link(struct inode *dir, const struct qstr *name, * consistency more. */ if (current != F2FS_I(dir)->task) { - de = __f2fs_find_entry(dir, &fname, &folio); + de = __f2fs_find_entry(dir, &fname, &dentry_blk); F2FS_I(dir)->task = NULL; } if (de) { - f2fs_folio_put(folio, false); + f2fs_put_dentry_block(dentry_blk, false); err = -EEXIST; - } else if (IS_ERR(folio)) { - err = PTR_ERR(folio); + } else if (IS_ERR(dentry_blk)) { + err = PTR_ERR(dentry_blk); } else { err = f2fs_add_dentry(dir, &fname, inode, ino, mode); } @@ -839,16 +850,16 @@ int f2fs_do_add_link(struct inode *dir, const struct qstr *name, int f2fs_do_tmpfile(struct inode *inode, struct inode *dir, struct f2fs_filename *fname) { - struct folio *folio; + struct f2fs_cached_block *ientry; int err = 0; f2fs_down_write(&F2FS_I(inode)->i_sem); - folio = f2fs_init_inode_metadata(inode, dir, fname, NULL); - if (IS_ERR(folio)) { - err = PTR_ERR(folio); + ientry = f2fs_init_inode_metadata(inode, dir, fname, NULL); + if (IS_ERR(ientry)) { + err = PTR_ERR(ientry); goto fail; } - f2fs_folio_put(folio, true); + f2fs_put_cache(ientry, true); clear_inode_flag(inode, FI_NEW_INODE); f2fs_update_time(F2FS_I_SB(inode), REQ_TIME); @@ -884,13 +895,14 @@ void f2fs_drop_nlink(struct inode *dir, struct inode *inode) * It only removes the dentry from the dentry page, corresponding name * entry in name page does not need to be touched during deletion. */ -void f2fs_delete_entry(struct f2fs_dir_entry *dentry, struct folio *folio, +void f2fs_delete_entry(struct f2fs_dir_entry *dentry, void *dentry_block, struct inode *dir, struct inode *inode) { struct f2fs_dentry_block *dentry_blk; + struct folio *folio; unsigned int bit_pos; int slots = GET_DENTRY_SLOTS(le16_to_cpu(dentry->name_len)); - pgoff_t index = folio->index; + pgoff_t index; int i; f2fs_update_time(F2FS_I_SB(dir), REQ_TIME); @@ -899,10 +911,14 @@ void f2fs_delete_entry(struct f2fs_dir_entry *dentry, struct folio *folio, f2fs_add_ino_entry(F2FS_I_SB(dir), dir->i_ino, TRANS_DIR_INO); if (f2fs_has_inline_dentry(dir)) - return f2fs_delete_inline_entry(dentry, folio, dir, inode); + return f2fs_delete_inline_entry(dentry, + f2fs_dentry_cache(dentry_block), dir, inode); + + folio = f2fs_dentry_folio(dentry_block); + index = folio->index; folio_lock(folio); - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); dentry_blk = folio_address(folio); bit_pos = dentry - dentry_blk->dentry; diff --git a/fs/f2fs/extent_cache.c b/fs/f2fs/extent_cache.c index 37cf9fa8d537..e52527c68c02 100644 --- a/fs/f2fs/extent_cache.c +++ b/fs/f2fs/extent_cache.c @@ -20,10 +20,10 @@ #include "segment.h" #include <trace/events/f2fs.h> -bool sanity_check_extent_cache(struct inode *inode, struct folio *ifolio) +bool sanity_check_extent_cache(struct inode *inode, struct f2fs_cached_block *ientry) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - struct f2fs_extent *i_ext = &F2FS_INODE(ifolio)->i_ext; + struct f2fs_extent *i_ext = &F2FS_INODE(ientry)->i_ext; struct extent_info ei; int devi; @@ -416,11 +416,11 @@ static void __drop_largest_extent(struct extent_tree *et, } } -void f2fs_init_read_extent_tree(struct inode *inode, struct folio *ifolio) +void f2fs_init_read_extent_tree(struct inode *inode, struct f2fs_cached_block *ientry) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct extent_tree_info *eti = &sbi->extent_tree[EX_READ]; - struct f2fs_extent *i_ext = &F2FS_INODE(ifolio)->i_ext; + struct f2fs_extent *i_ext = &F2FS_INODE(ientry)->i_ext; struct extent_tree *et; struct extent_node *en; struct extent_info ei = {0}; @@ -428,9 +428,9 @@ void f2fs_init_read_extent_tree(struct inode *inode, struct folio *ifolio) if (!__may_extent_tree(inode, EX_READ)) { /* drop largest read extent */ if (i_ext->len) { - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_cache_wait_writeback(ientry); i_ext->len = 0; - folio_mark_dirty(ifolio); + f2fs_mark_cache_dirty(ientry); } set_inode_flag(inode, FI_NO_EXTENT); return; @@ -956,7 +956,7 @@ static void __update_extent_cache(struct dnode_of_data *dn, enum extent_type typ if (!__may_extent_tree(dn->inode, type)) return; - ei.fofs = f2fs_start_bidx_of_node(ofs_of_node(dn->node_folio), dn->inode) + + ei.fofs = f2fs_start_bidx_of_node(ofs_of_node(dn->node_entry), dn->inode) + dn->ofs_in_node; ei.len = 1; diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 4a811e9d325b..4dd165ac05c9 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -416,7 +416,7 @@ struct inode_entry { struct fsync_node_entry { struct list_head list; /* list head */ - struct folio *folio; /* warm node folio pointer */ + struct f2fs_cached_block *entry; /* warm node cache entry pointer */ unsigned int seq_id; /* sequence id */ }; @@ -1123,11 +1123,11 @@ struct f2fs_nm_info { */ struct dnode_of_data { struct inode *inode; /* vfs inode pointer */ - struct folio *inode_folio; /* its inode folio, NULL is possible */ - struct folio *node_folio; /* cached direct node folio */ + struct f2fs_cached_block *inode_entry; /* generic cache inode entry */ + struct f2fs_cached_block *node_entry; /* generic cache node entry */ nid_t nid; /* node id of the direct node block */ unsigned int ofs_in_node; /* data offset in the node page */ - bool inode_folio_locked; /* inode folio is locked or not */ + bool inode_entry_locked; /* inode entry is locked or not */ bool node_changed; /* is node block changed */ char cur_level; /* level of hole node page */ char max_level; /* level of current page located */ @@ -1135,12 +1135,12 @@ struct dnode_of_data { }; static inline void set_new_dnode(struct dnode_of_data *dn, struct inode *inode, - struct folio *ifolio, struct folio *nfolio, nid_t nid) + struct f2fs_cached_block *ientry, struct f2fs_cached_block *nentry, nid_t nid) { memset(dn, 0, sizeof(*dn)); dn->inode = inode; - dn->inode_folio = ifolio; - dn->node_folio = nfolio; + dn->inode_entry = ientry; + dn->node_entry = nentry; dn->nid = nid; } @@ -1622,10 +1622,9 @@ static inline void f2fs_clear_bit(unsigned int nr, char *addr); * | bit0 = 1 | bit1 | bit2 | ... | bit MAX | private data .... | * bit 0 PAGE_PRIVATE_NOT_POINTER * bit 1 PAGE_PRIVATE_ONGOING_MIGRATION - * bit 2 PAGE_PRIVATE_INLINE_INODE - * bit 3 PAGE_PRIVATE_REF_RESOURCE - * bit 4 PAGE_PRIVATE_ATOMIC_WRITE - * bit 5- f2fs private data + * bit 2 PAGE_PRIVATE_REF_RESOURCE + * bit 3 PAGE_PRIVATE_ATOMIC_WRITE + * bit 4- f2fs private data * * Layout B: lowest bit should be 0 * page.private is a wrapped pointer. @@ -1633,7 +1632,6 @@ static inline void f2fs_clear_bit(unsigned int nr, char *addr); enum { PAGE_PRIVATE_NOT_POINTER, /* private contains non-pointer data */ PAGE_PRIVATE_ONGOING_MIGRATION, /* data page which is on-going migrating */ - PAGE_PRIVATE_INLINE_INODE, /* inode page contains inline data */ PAGE_PRIVATE_REF_RESOURCE, /* dirty page has referenced resources */ PAGE_PRIVATE_ATOMIC_WRITE, /* data page from atomic write path */ PAGE_PRIVATE_MAX @@ -1810,7 +1808,6 @@ struct f2fs_sb_info { /* for node-related operations */ struct f2fs_nm_info *nm_info; /* node manager */ - struct inode *node_inode; /* cache node blocks */ /* for segment-related operations */ struct f2fs_sm_info *sm_info; /* segment manager */ @@ -2283,14 +2280,14 @@ static inline struct f2fs_checkpoint *F2FS_CKPT(struct f2fs_sb_info *sbi) return (struct f2fs_checkpoint *)(sbi->ckpt); } -static inline struct f2fs_node *F2FS_NODE(const struct folio *folio) +static inline struct f2fs_node *F2FS_NODE(const struct f2fs_cached_block *entry) { - return (struct f2fs_node *)folio_address(folio); + return (struct f2fs_node *)CACHED_NODE(entry); } -static inline struct f2fs_inode *F2FS_INODE(const struct folio *folio) +static inline struct f2fs_inode *F2FS_INODE(const struct f2fs_cached_block *entry) { - return &((struct f2fs_node *)folio_address(folio))->i; + return &CACHED_NODE(entry)->i; } static inline struct f2fs_nm_info *NM_I(struct f2fs_sb_info *sbi) @@ -2323,14 +2320,9 @@ static inline bool f2fs_is_meta_cache(struct f2fs_cached_block *entry) return entry->cache && entry->cache == META_CACHE(entry->cache->sbi); } -static inline struct address_space *NODE_MAPPING(struct f2fs_sb_info *sbi) +static inline bool f2fs_is_node_cache(struct f2fs_cached_block *entry) { - return sbi->node_inode->i_mapping; -} - -static inline bool is_node_folio(struct folio *folio) -{ - return folio->mapping == NODE_MAPPING(F2FS_F_SB(folio)); + return entry->cache && entry->cache == NODE_CACHE(entry->cache->sbi); } static inline struct f2fs_bio *F2FS_BIO(struct bio *bio) @@ -2753,17 +2745,14 @@ static inline void clear_page_private_##name(struct page *page) \ } PAGE_PRIVATE_GET_FUNC(nonpointer, NOT_POINTER); -PAGE_PRIVATE_GET_FUNC(inline, INLINE_INODE); PAGE_PRIVATE_GET_FUNC(gcing, ONGOING_MIGRATION); PAGE_PRIVATE_GET_FUNC(atomic, ATOMIC_WRITE); PAGE_PRIVATE_SET_FUNC(reference, REF_RESOURCE); -PAGE_PRIVATE_SET_FUNC(inline, INLINE_INODE); PAGE_PRIVATE_SET_FUNC(gcing, ONGOING_MIGRATION); PAGE_PRIVATE_SET_FUNC(atomic, ATOMIC_WRITE); PAGE_PRIVATE_CLEAR_FUNC(reference, REF_RESOURCE); -PAGE_PRIVATE_CLEAR_FUNC(inline, INLINE_INODE); PAGE_PRIVATE_CLEAR_FUNC(gcing, ONGOING_MIGRATION); PAGE_PRIVATE_CLEAR_FUNC(atomic, ATOMIC_WRITE); @@ -3172,14 +3161,51 @@ static inline void f2fs_put_page(struct page *page, bool unlock) f2fs_folio_put(page_folio(page), unlock); } +#define F2FS_DENTRY_TAG_CACHE 1UL +#define F2FS_DENTRY_TAG_MASK 1UL + +static inline void *f2fs_cache_make_dentry_block(struct f2fs_cached_block *entry) +{ + if (IS_ERR_OR_NULL(entry)) + return entry; + return (void *)((unsigned long)entry | F2FS_DENTRY_TAG_CACHE); +} + +static inline bool f2fs_dentry_is_cache(void *dentry_block) +{ + return ((unsigned long)dentry_block & F2FS_DENTRY_TAG_MASK) == + F2FS_DENTRY_TAG_CACHE; +} + +static inline struct f2fs_cached_block *f2fs_dentry_cache(void *dentry_block) +{ + return (struct f2fs_cached_block *) + ((unsigned long)dentry_block & ~F2FS_DENTRY_TAG_MASK); +} + +static inline struct folio *f2fs_dentry_folio(void *dentry_block) +{ + return (struct folio *)dentry_block; +} + +static inline void f2fs_put_dentry_block(void *dentry_block, bool unlock) +{ + if (IS_ERR_OR_NULL(dentry_block)) + return; + if (f2fs_dentry_is_cache(dentry_block)) + f2fs_put_cache(f2fs_dentry_cache(dentry_block), unlock); + else + f2fs_folio_put(f2fs_dentry_folio(dentry_block), unlock); +} + static inline void f2fs_put_dnode(struct dnode_of_data *dn) { - if (dn->node_folio) - f2fs_folio_put(dn->node_folio, true); - if (dn->inode_folio && dn->node_folio != dn->inode_folio) - f2fs_folio_put(dn->inode_folio, false); - dn->node_folio = NULL; - dn->inode_folio = NULL; + if (dn->node_entry) + f2fs_put_cache(dn->node_entry, true); + if (dn->inode_entry && dn->node_entry != dn->inode_entry) + f2fs_put_cache(dn->inode_entry, false); + dn->node_entry = NULL; + dn->inode_entry = NULL; } static inline struct kmem_cache *f2fs_kmem_cache_create(const char *name, @@ -3270,9 +3296,9 @@ static inline void f2fs_radix_tree_insert(struct radix_tree_root *root, #define RAW_IS_INODE(p) ((p)->footer.nid == (p)->footer.ino) -static inline bool IS_INODE(const struct folio *folio) +static inline bool IS_INODE(const struct f2fs_cached_block *entry) { - struct f2fs_node *p = F2FS_NODE(folio); + struct f2fs_node *p = F2FS_NODE(entry); return RAW_IS_INODE(p); } @@ -3289,32 +3315,34 @@ static inline __le32 *blkaddr_in_node(struct f2fs_node *node) } static inline int f2fs_has_extra_attr(struct inode *inode); + static inline unsigned int get_dnode_base(struct inode *inode, - struct folio *node_folio) + const struct f2fs_cached_block *entry) { - if (!IS_INODE(node_folio)) + if (!IS_INODE(entry)) return 0; return inode ? get_extra_isize(inode) : - offset_in_addr(&F2FS_NODE(node_folio)->i); + offset_in_addr(&CACHED_NODE(entry)->i); } static inline __le32 *get_dnode_addr(struct inode *inode, - struct folio *node_folio) + const struct f2fs_cached_block *entry) { - return blkaddr_in_node(F2FS_NODE(node_folio)) + - get_dnode_base(inode, node_folio); + return blkaddr_in_node(CACHED_NODE(entry)) + + get_dnode_base(inode, entry); } static inline block_t data_blkaddr(struct inode *inode, - struct folio *node_folio, unsigned int offset) + const struct f2fs_cached_block *entry, + unsigned int offset) { - return le32_to_cpu(*(get_dnode_addr(inode, node_folio) + offset)); + return le32_to_cpu(*(get_dnode_addr(inode, entry) + offset)); } static inline block_t f2fs_data_blkaddr(struct dnode_of_data *dn) { - return data_blkaddr(dn->inode, dn->node_folio, dn->ofs_in_node); + return data_blkaddr(dn->inode, dn->node_entry, dn->ofs_in_node); } static inline int f2fs_test_bit(unsigned int nr, char *addr) @@ -3625,10 +3653,10 @@ static inline unsigned int addrs_per_page(struct inode *inode, return addrs; } -static inline -void *inline_xattr_addr(struct inode *inode, const struct folio *folio) +static inline void *inline_xattr_addr(struct inode *inode, + const struct f2fs_cached_block *entry) { - struct f2fs_inode *ri = F2FS_INODE(folio); + struct f2fs_inode *ri = F2FS_INODE(entry); return (void *)&(ri->i_addr[DEF_ADDRS_PER_INODE - get_inline_xattr_addrs(inode)]); @@ -3675,9 +3703,10 @@ static inline bool f2fs_is_cow_file(struct inode *inode) return is_inode_flag_set(inode, FI_COW_FILE); } -static inline void *inline_data_addr(struct inode *inode, struct folio *folio) +static inline void *inline_data_addr(struct inode *inode, + const struct f2fs_cached_block *entry) { - __le32 *addr = get_dnode_addr(inode, folio); + __le32 *addr = get_dnode_addr(inode, entry); return (void *)(addr + DEF_INLINE_RESERVED_SIZE); } @@ -3887,12 +3916,12 @@ int f2fs_pin_file_control(struct inode *inode, bool inc); * inode.c */ void f2fs_set_inode_flags(struct inode *inode); -bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct folio *folio); -void f2fs_inode_chksum_set(struct f2fs_sb_info *sbi, struct folio *folio); +bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry); +void f2fs_inode_chksum_set(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry); struct inode *f2fs_iget(struct super_block *sb, unsigned long ino); struct inode *f2fs_iget_retry(struct super_block *sb, unsigned long ino); int f2fs_try_to_free_nats(struct f2fs_sb_info *sbi, int nr_shrink); -void f2fs_update_inode(struct inode *inode, struct folio *node_folio); +void f2fs_update_inode(struct inode *inode, struct f2fs_cached_block *entry); void f2fs_update_inode_page(struct inode *inode); int f2fs_write_inode(struct inode *inode, struct writeback_control *wbc); void f2fs_remove_donate_inode(struct inode *inode); @@ -3941,22 +3970,22 @@ int f2fs_fill_dentries(struct dir_context *ctx, struct f2fs_dentry_ptr *d, unsigned int start_pos, struct fscrypt_str *fstr); void f2fs_do_make_empty_dir(struct inode *inode, struct inode *parent, struct f2fs_dentry_ptr *d); -struct folio *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir, - const struct f2fs_filename *fname, struct folio *dfolio); +struct f2fs_cached_block *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir, + const struct f2fs_filename *fname, struct f2fs_cached_block *dentry); void f2fs_update_parent_metadata(struct inode *dir, struct inode *inode, unsigned int current_depth); int f2fs_room_for_filename(const void *bitmap, int slots, int max_slots); void f2fs_drop_nlink(struct inode *dir, struct inode *inode); struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir, - const struct f2fs_filename *fname, struct folio **res_folio); + const struct f2fs_filename *fname, void **dentry_block); struct f2fs_dir_entry *f2fs_find_entry(struct inode *dir, - const struct qstr *child, struct folio **res_folio); -struct f2fs_dir_entry *f2fs_parent_dir(struct inode *dir, struct folio **f); + const struct qstr *child, void **dentry_block); +struct f2fs_dir_entry *f2fs_parent_dir(struct inode *dir, void **dentry_block); ino_t f2fs_inode_by_name(struct inode *dir, const struct qstr *qstr, - struct folio **folio); + void **dentry_block); void f2fs_set_link(struct inode *dir, struct f2fs_dir_entry *de, - struct folio *folio, struct inode *inode); -bool f2fs_has_enough_room(struct inode *dir, struct folio *ifolio, + void *dentry_blk, struct inode *inode); +bool f2fs_has_enough_room(struct inode *dir, struct f2fs_cached_block *ientry, const struct f2fs_filename *fname); void f2fs_update_dentry(nid_t ino, umode_t mode, struct f2fs_dentry_ptr *d, const struct fscrypt_str *name, f2fs_hash_t name_hash, @@ -3967,7 +3996,7 @@ int f2fs_add_dentry(struct inode *dir, const struct f2fs_filename *fname, struct inode *inode, nid_t ino, umode_t mode); int f2fs_do_add_link(struct inode *dir, const struct qstr *name, struct inode *inode, nid_t ino, umode_t mode); -void f2fs_delete_entry(struct f2fs_dir_entry *dentry, struct folio *folio, +void f2fs_delete_entry(struct f2fs_dir_entry *dentry, void *dentry_blk, struct inode *dir, struct inode *inode); int f2fs_do_tmpfile(struct inode *inode, struct inode *dir, struct f2fs_filename *fname); @@ -4010,9 +4039,9 @@ enum node_type; int f2fs_check_nid_range(struct f2fs_sb_info *sbi, nid_t nid); bool f2fs_available_free_memory(struct f2fs_sb_info *sbi, int type); -bool f2fs_in_warm_node_list(struct folio *folio); +bool f2fs_in_warm_node_list(struct f2fs_cached_block *entry); void f2fs_init_fsync_node_info(struct f2fs_sb_info *sbi); -void f2fs_del_fsync_node_entry(struct f2fs_sb_info *sbi, struct folio *folio); +void f2fs_del_fsync_node_entry(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry); void f2fs_reset_fsync_node_info(struct f2fs_sb_info *sbi); bool f2fs_need_dentry_mark(struct f2fs_sb_info *sbi, nid_t nid); bool f2fs_is_checkpointed_node(struct f2fs_sb_info *sbi, nid_t nid); @@ -4023,29 +4052,28 @@ pgoff_t f2fs_get_next_page_offset(struct dnode_of_data *dn, pgoff_t pgofs); int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode); int f2fs_truncate_inode_blocks(struct inode *inode, pgoff_t from); int f2fs_truncate_xattr_node(struct inode *inode); -int f2fs_wait_on_node_pages_writeback(struct f2fs_sb_info *sbi, +int f2fs_wait_on_node_caches_writeback(struct f2fs_sb_info *sbi, unsigned int seq_id); +int f2fs_write_node_caches(struct f2fs_sb_info *sbi); int f2fs_remove_inode_page(struct inode *inode); -struct folio *f2fs_new_inode_folio(struct inode *inode); -struct folio *f2fs_new_node_folio(struct dnode_of_data *dn, unsigned int ofs); +struct f2fs_cached_block *f2fs_new_inode_cache(struct inode *inode); +struct f2fs_cached_block *f2fs_new_node_cache(struct dnode_of_data *dn, unsigned int ofs); void f2fs_ra_node_page(struct f2fs_sb_info *sbi, nid_t nid); -struct folio *f2fs_get_node_folio(struct f2fs_sb_info *sbi, pgoff_t nid, +struct f2fs_cached_block *f2fs_get_node_cache(struct f2fs_sb_info *sbi, pgoff_t nid, enum node_type node_type); int f2fs_sanity_check_node_footer(struct f2fs_sb_info *sbi, - struct folio *folio, pgoff_t nid, + struct f2fs_cached_block *entry, pgoff_t nid, enum node_type ntype, bool in_irq); -struct folio *f2fs_get_inode_folio(struct f2fs_sb_info *sbi, pgoff_t ino); -struct folio *f2fs_get_xnode_folio(struct f2fs_sb_info *sbi, pgoff_t xnid); -int f2fs_write_single_node_folio(struct folio *node_folio, int sync_mode, +struct f2fs_cached_block *f2fs_get_inode_cache(struct f2fs_sb_info *sbi, pgoff_t ino); +struct f2fs_cached_block *f2fs_get_xnode_cache(struct f2fs_sb_info *sbi, pgoff_t xnid); +int f2fs_write_node_cache(struct f2fs_cached_block *entry, int sync_mode, bool mark_dirty, enum iostat_type io_type); -int f2fs_move_node_folio(struct folio *node_folio, int gc_type); +int f2fs_move_node_cache(struct f2fs_cached_block *entry, int gc_type); void f2fs_flush_inline_data(struct f2fs_sb_info *sbi); -int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode, - struct writeback_control *wbc, bool atomic, - unsigned int *seq_id); -int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, - struct writeback_control *wbc, - bool do_balance, enum iostat_type io_type); +int f2fs_fsync_node_caches(struct f2fs_sb_info *sbi, struct inode *inode, + bool atomic, unsigned int *seq_id); +int f2fs_writeback_node_caches(struct f2fs_sb_info *sbi, long nr_to_write, + bool sync, bool do_balance, enum iostat_type io_type); int f2fs_build_free_nids(struct f2fs_sb_info *sbi, bool sync, bool mount); bool f2fs_alloc_nid(struct f2fs_sb_info *sbi, nid_t *nid); void f2fs_alloc_nid_done(struct f2fs_sb_info *sbi, nid_t nid); @@ -4124,14 +4152,13 @@ void f2fs_replace_block(struct f2fs_sb_info *sbi, struct dnode_of_data *dn, bool recover_newaddr); enum temp_type f2fs_get_segment_temp(struct f2fs_sb_info *sbi, enum log_type seg_type); -int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, struct folio *folio, +int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, block_t old_blkaddr, block_t *new_blkaddr, struct f2fs_summary *sum, int type, struct f2fs_io_info *fio); void f2fs_update_device_state(struct f2fs_sb_info *sbi, nid_t ino, block_t blkaddr, unsigned int blkcnt); -void f2fs_folio_wait_writeback(struct folio *folio, enum page_type type, - bool ordered, bool locked); +void f2fs_folio_wait_writeback(struct folio *folio, bool ordered, bool locked); void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr); void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr, block_t len); @@ -4252,14 +4279,14 @@ void f2fs_destroy_bio_entry_cache(void); void f2fs_submit_read_bio(struct f2fs_sb_info *sbi, struct bio *bio, enum page_type type); int f2fs_init_write_merge_io(struct f2fs_sb_info *sbi); -void f2fs_submit_merged_write(struct f2fs_sb_info *sbi, enum page_type type); void f2fs_submit_merged_write_cond(struct f2fs_sb_info *sbi, - struct inode *inode, struct folio *folio, - nid_t ino, enum page_type type); + struct inode *inode, struct folio *folio); void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, - struct folio *folio, enum page_type type); -bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, - enum page_type type); + struct folio *folio); +bool f2fs_submit_merged_write_cache(struct f2fs_sb_info *sbi, + struct f2fs_cached_block *entry, + nid_t ino, enum page_type type); +void f2fs_submit_merged_write(struct f2fs_sb_info *sbi, enum page_type type); void f2fs_submit_merged_ipu_write(struct f2fs_sb_info *sbi, struct bio **bio, struct folio *folio); void f2fs_submit_all_merged_ipu_writes(struct f2fs_sb_info *sbi); @@ -4285,7 +4312,7 @@ struct folio *f2fs_find_data_folio(struct inode *inode, pgoff_t index, struct folio *f2fs_get_lock_data_folio(struct inode *inode, pgoff_t index, bool for_write); struct folio *f2fs_get_new_data_folio(struct inode *inode, - struct folio *ifolio, pgoff_t index, bool new_i_size); + struct f2fs_cached_block *ientry, pgoff_t index, bool new_i_size); int f2fs_do_write_data_page(struct f2fs_io_info *fio); int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map, int flag); int f2fs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, @@ -4398,7 +4425,7 @@ struct f2fs_stat_info { unsigned int bimodal, avg_vblocks; int util_free, util_valid, util_invalid; int rsvd_segs, overp_segs; - int dirty_count, node_pages, meta_caches, compress_pages; + int dirty_count, node_caches, meta_caches, compress_pages; int compress_page_hit; int prefree_count, free_segs, free_secs; int cp_call_count[MAX_CALL_TYPE], cp_count; @@ -4599,7 +4626,6 @@ extern const struct file_operations f2fs_dir_operations; extern const struct file_operations f2fs_file_operations; extern const struct inode_operations f2fs_file_inode_operations; extern const struct address_space_operations f2fs_dblock_aops; -extern const struct address_space_operations f2fs_node_aops; extern const struct inode_operations f2fs_dir_inode_operations; extern const struct inode_operations f2fs_symlink_inode_operations; extern const struct inode_operations f2fs_encrypted_symlink_inode_operations; @@ -4610,10 +4636,10 @@ extern struct kmem_cache *f2fs_inode_entry_slab; * inline.c */ bool f2fs_may_inline_data(struct inode *inode); -bool f2fs_sanity_check_inline_data(struct inode *inode, struct folio *ifolio); +bool f2fs_sanity_check_inline_data(struct inode *inode, struct f2fs_cached_block *ientry); bool f2fs_may_inline_dentry(struct inode *inode); -void f2fs_do_read_inline_data(struct folio *folio, struct folio *ifolio); -void f2fs_truncate_inline_inode(struct inode *inode, struct folio *ifolio, +void f2fs_do_read_inline_data(struct folio *folio, struct f2fs_cached_block *ientry); +void f2fs_truncate_inline_inode(struct inode *inode, struct f2fs_cached_block *ientry, u64 from); int f2fs_read_inline_data(struct inode *inode, struct folio *folio); int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio); @@ -4622,14 +4648,15 @@ int f2fs_try_convert_inline_dir(struct inode *dir, struct dentry *dentry); int f2fs_write_inline_data(struct inode *inode, struct folio *folio); int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entry); struct f2fs_dir_entry *f2fs_find_in_inline_dir(struct inode *dir, - const struct f2fs_filename *fname, struct folio **res_folio, + const struct f2fs_filename *fname, void **dentry_block, bool use_hash); int f2fs_make_empty_inline_dir(struct inode *inode, struct inode *parent, - struct folio *ifolio); + struct f2fs_cached_block *ientry); int f2fs_add_inline_entry(struct inode *dir, const struct f2fs_filename *fname, struct inode *inode, nid_t ino, umode_t mode); void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry, - struct folio *folio, struct inode *dir, struct inode *inode); + struct f2fs_cached_block *ientry, struct inode *dir, + struct inode *inode); bool f2fs_empty_inline_dir(struct inode *dir); int f2fs_read_inline_dir(struct file *file, struct dir_context *ctx, struct fscrypt_str *fstr); @@ -4652,7 +4679,7 @@ void f2fs_leave_shrinker(struct f2fs_sb_info *sbi); /* * extent_cache.c */ -bool sanity_check_extent_cache(struct inode *inode, struct folio *ifolio); +bool sanity_check_extent_cache(struct inode *inode, struct f2fs_cached_block *ientry); void f2fs_init_extent_tree(struct inode *inode); void f2fs_drop_extent_tree(struct inode *inode); void f2fs_destroy_extent_node(struct inode *inode); @@ -4662,7 +4689,7 @@ int __init f2fs_create_extent_cache(void); void f2fs_destroy_extent_cache(void); /* read extent cache ops */ -void f2fs_init_read_extent_tree(struct inode *inode, struct folio *ifolio); +void f2fs_init_read_extent_tree(struct inode *inode, struct f2fs_cached_block *ientry); bool f2fs_lookup_read_extent_cache(struct inode *inode, pgoff_t pgofs, struct extent_info *ei); bool f2fs_lookup_read_extent_cache_block(struct inode *inode, pgoff_t index, diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c index 92daa41dd96d..71d124f07abd 100644 --- a/fs/f2fs/file.c +++ b/fs/f2fs/file.c @@ -212,7 +212,7 @@ static vm_fault_t f2fs_vm_page_mkwrite(struct vm_fault *vmf) goto out_sem; } - f2fs_folio_wait_writeback(folio, DATA, false, true); + f2fs_folio_wait_writeback(folio, false, true); /* wait for GCed page writeback via generic cache */ f2fs_wait_on_block_writeback(inode, dn.data_blkaddr); @@ -307,13 +307,13 @@ static inline enum cp_reason_type need_do_checkpoint(struct inode *inode) static bool need_inode_page_update(struct f2fs_sb_info *sbi, nid_t ino) { - struct folio *i = filemap_get_folio(NODE_MAPPING(sbi), ino); + struct f2fs_cached_block *entry = f2fs_find_node_cache(sbi, ino); bool ret = false; /* But we need to avoid that there are some inode updates */ - if ((!IS_ERR(i) && folio_test_dirty(i)) || + if ((!IS_ERR(entry) && f2fs_cache_test_dirty(entry)) || f2fs_need_inode_block_update(sbi, ino)) ret = true; - f2fs_folio_put(i, false); + f2fs_put_cache(entry, false); return ret; } @@ -339,10 +339,6 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end, nid_t ino = inode->i_ino; int ret = 0; enum cp_reason_type cp_reason = 0; - struct writeback_control wbc = { - .sync_mode = WB_SYNC_ALL, - .nr_to_write = LONG_MAX, - }; unsigned int seq_id = 0; if (unlikely(f2fs_readonly(inode->i_sb))) @@ -421,7 +417,7 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end, } sync_nodes: atomic_inc(&sbi->wb_sync_req[NODE]); - ret = f2fs_fsync_node_pages(sbi, inode, &wbc, atomic, &seq_id); + ret = f2fs_fsync_node_caches(sbi, inode, atomic, &seq_id); atomic_dec(&sbi->wb_sync_req[NODE]); if (ret) goto out; @@ -447,7 +443,7 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end, * given fsync mark. */ if (!atomic) { - ret = f2fs_wait_on_node_pages_writeback(sbi, seq_id); + ret = f2fs_wait_on_node_caches_writeback(sbi, seq_id); if (ret) goto out; } @@ -484,7 +480,7 @@ static bool __found_offset(struct address_space *mapping, bool compressed_cluster = false; if (f2fs_compressed_file(inode)) { - block_t first_blkaddr = data_blkaddr(dn->inode, dn->node_folio, + block_t first_blkaddr = data_blkaddr(dn->inode, dn->node_entry, ALIGN_DOWN(dn->ofs_in_node, F2FS_I(inode)->i_cluster_size)); compressed_cluster = first_blkaddr == COMPRESS_ADDR; @@ -554,7 +550,7 @@ static loff_t f2fs_seek_block(struct file *file, loff_t offset, int whence) } } - end_offset = ADDRS_PER_PAGE(dn.node_folio, inode); + end_offset = ADDRS_PER_PAGE(dn.node_entry, inode); /* find data/hole in dnode block */ for (; dn.ofs_in_node < end_offset; @@ -716,7 +712,7 @@ void f2fs_truncate_data_blocks_range(struct dnode_of_data *dn, int count) block_t blkstart; int blklen = 0; - addr = get_dnode_addr(dn->inode, dn->node_folio) + ofs; + addr = get_dnode_addr(dn->inode, dn->node_entry) + ofs; blkstart = le32_to_cpu(*addr); /* Assumption: truncation starts with cluster */ @@ -780,7 +776,7 @@ void f2fs_truncate_data_blocks_range(struct dnode_of_data *dn, int count) * once we invalidate valid blkaddr in range [ofs, ofs + count], * we will invalidate all blkaddr in the whole range. */ - fofs = f2fs_start_bidx_of_node(ofs_of_node(dn->node_folio), + fofs = f2fs_start_bidx_of_node(ofs_of_node(dn->node_entry), dn->inode) + ofs; f2fs_update_read_extent_cache_range(dn, fofs, 0, len); f2fs_update_age_extent_cache_range(dn, fofs, len); @@ -818,7 +814,7 @@ static int truncate_partial_data_page(struct inode *inode, u64 from, if (IS_ERR(folio)) return PTR_ERR(folio) == -ENOENT ? 0 : PTR_ERR(folio); truncate_out: - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); folio_zero_segment(folio, offset, folio_size(folio)); /* An encrypted inode should have a key and truncate the last page. */ @@ -836,7 +832,7 @@ int f2fs_do_truncate_blocks(struct inode *inode, u64 from, bool lock) struct f2fs_lock_context lc; pgoff_t free_from; int count = 0, err = 0; - struct folio *ifolio; + struct f2fs_cached_block *ientry; bool truncate_page = false; trace_f2fs_truncate_blocks_enter(inode, from); @@ -854,9 +850,9 @@ int f2fs_do_truncate_blocks(struct inode *inode, u64 from, bool lock) if (lock) f2fs_lock_op(sbi, &lc); - ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(ifolio)) { - err = PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(ientry)) { + err = PTR_ERR(ientry); goto out; } @@ -875,18 +871,18 @@ int f2fs_do_truncate_blocks(struct inode *inode, u64 from, bool lock) f2fs_drop_extent_tree(inode); - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); goto out; } if (f2fs_has_inline_data(inode)) { - f2fs_truncate_inline_inode(inode, ifolio, from); - f2fs_folio_put(ifolio, true); + f2fs_truncate_inline_inode(inode, ientry, from); + f2fs_put_cache(ientry, true); truncate_page = true; goto out; } - set_new_dnode(&dn, inode, ifolio, NULL, 0); + set_new_dnode(&dn, inode, ientry, NULL, 0); err = f2fs_get_dnode_of_data(&dn, free_from, LOOKUP_NODE_RA); if (err) { if (err == -ENOENT) @@ -894,12 +890,12 @@ int f2fs_do_truncate_blocks(struct inode *inode, u64 from, bool lock) goto out; } - count = ADDRS_PER_PAGE(dn.node_folio, inode); + count = ADDRS_PER_PAGE(dn.node_entry, inode); count -= dn.ofs_in_node; f2fs_bug_on(sbi, count < 0); - if (dn.ofs_in_node || IS_INODE(dn.node_folio)) { + if (dn.ofs_in_node || IS_INODE(dn.node_entry)) { f2fs_truncate_data_blocks_range(&dn, count); free_from += count; } @@ -1308,7 +1304,7 @@ static int fill_zero(struct inode *inode, pgoff_t index, if (IS_ERR(folio)) return PTR_ERR(folio); - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); folio_zero_range(folio, start, len); folio_mark_dirty(folio); f2fs_folio_put(folio, true); @@ -1334,7 +1330,7 @@ int f2fs_truncate_hole(struct inode *inode, pgoff_t pg_start, pgoff_t pg_end) return err; } - end_offset = ADDRS_PER_PAGE(dn.node_folio, inode); + end_offset = ADDRS_PER_PAGE(dn.node_entry, inode); count = min(end_offset - dn.ofs_in_node, pg_end - pg_start); f2fs_bug_on(F2FS_I_SB(inode), count == 0 || count > end_offset); @@ -1434,7 +1430,7 @@ static int __read_out_blkaddrs(struct inode *inode, block_t *blkaddr, goto next; } - done = min((pgoff_t)ADDRS_PER_PAGE(dn.node_folio, inode) - + done = min((pgoff_t)ADDRS_PER_PAGE(dn.node_entry, inode) - dn.ofs_in_node, len); for (i = 0; i < done; i++, blkaddr++, do_replace++, dn.ofs_in_node++) { *blkaddr = f2fs_data_blkaddr(&dn); @@ -1523,7 +1519,7 @@ static int __clone_blkaddrs(struct inode *src_inode, struct inode *dst_inode, } ilen = min((pgoff_t) - ADDRS_PER_PAGE(dn.node_folio, dst_inode) - + ADDRS_PER_PAGE(dn.node_entry, dst_inode) - dn.ofs_in_node, len - i); do { dn.data_blkaddr = f2fs_data_blkaddr(&dn); @@ -1561,7 +1557,7 @@ static int __clone_blkaddrs(struct inode *src_inode, struct inode *dst_inode, return PTR_ERR(fdst); } - f2fs_folio_wait_writeback(fdst, DATA, true, true); + f2fs_folio_wait_writeback(fdst, true, true); memcpy_folio(fdst, 0, fsrc, 0, PAGE_SIZE); folio_mark_dirty(fdst); @@ -1830,7 +1826,7 @@ static int f2fs_zero_range(struct inode *inode, loff_t offset, loff_t len, goto out; } - end_offset = ADDRS_PER_PAGE(dn.node_folio, inode); + end_offset = ADDRS_PER_PAGE(dn.node_entry, inode); end = min(pg_end, end_offset - dn.ofs_in_node + index); ret = f2fs_do_zero_range(&dn, index, end); @@ -3128,7 +3124,7 @@ static int f2fs_defragment_range(struct f2fs_sb_info *sbi, goto clear_out; } - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); folio_mark_dirty(folio); folio_set_f2fs_gcing(folio); @@ -4162,7 +4158,7 @@ static int release_compress_blocks(struct dnode_of_data *dn, pgoff_t count) int i; for (i = 0; i < count; i++) { - blkaddr = data_blkaddr(dn->inode, dn->node_folio, + blkaddr = data_blkaddr(dn->inode, dn->node_entry, dn->ofs_in_node + i); if (!__is_valid_data_blkaddr(blkaddr)) @@ -4281,7 +4277,7 @@ static int f2fs_release_compress_blocks(struct file *filp, unsigned long arg) break; } - end_offset = ADDRS_PER_PAGE(dn.node_folio, inode); + end_offset = ADDRS_PER_PAGE(dn.node_entry, inode); count = min(end_offset - dn.ofs_in_node, last_idx - page_idx); count = round_up(count, fi->i_cluster_size); @@ -4332,7 +4328,7 @@ static int reserve_compress_blocks(struct dnode_of_data *dn, pgoff_t count, int i; for (i = 0; i < count; i++) { - blkaddr = data_blkaddr(dn->inode, dn->node_folio, + blkaddr = data_blkaddr(dn->inode, dn->node_entry, dn->ofs_in_node + i); if (!__is_valid_data_blkaddr(blkaddr)) @@ -4349,7 +4345,7 @@ static int reserve_compress_blocks(struct dnode_of_data *dn, pgoff_t count, int ret; for (i = 0; i < cluster_size; i++) { - blkaddr = data_blkaddr(dn->inode, dn->node_folio, + blkaddr = data_blkaddr(dn->inode, dn->node_entry, dn->ofs_in_node + i); if (i == 0) { @@ -4460,7 +4456,7 @@ static int f2fs_reserve_compress_blocks(struct file *filp, unsigned long arg) break; } - end_offset = ADDRS_PER_PAGE(dn.node_folio, inode); + end_offset = ADDRS_PER_PAGE(dn.node_entry, inode); count = min(end_offset - dn.ofs_in_node, last_idx - page_idx); count = round_up(count, fi->i_cluster_size); @@ -4626,7 +4622,7 @@ static int f2fs_sec_trim_file(struct file *filp, unsigned long arg) goto out; } - end_offset = ADDRS_PER_PAGE(dn.node_folio, inode); + end_offset = ADDRS_PER_PAGE(dn.node_entry, inode); count = min(end_offset - dn.ofs_in_node, pg_end - index); for (i = 0; i < count; i++, index++, dn.ofs_in_node++) { struct block_device *cur_bdev; @@ -4822,7 +4818,7 @@ static int redirty_blocks(struct inode *inode, pgoff_t page_idx, int len) /* It will never fail, when folio has pinned above */ f2fs_bug_on(F2FS_I_SB(inode), IS_ERR(folio)); - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); folio_mark_dirty(folio); folio_set_f2fs_gcing(folio); diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c index 54327cb2e27e..12787d0414ec 100644 --- a/fs/f2fs/gc.c +++ b/fs/f2fs/gc.c @@ -1054,7 +1054,7 @@ static int gc_node_segment(struct f2fs_sb_info *sbi, for (off = 0; off < usable_blks_in_seg; off++, entry++) { nid_t nid = le32_to_cpu(entry->nid); - struct folio *node_folio; + struct f2fs_cached_block *node_entry; struct node_info ni; int err; @@ -1077,27 +1077,27 @@ static int gc_node_segment(struct f2fs_sb_info *sbi, } /* phase == 2 */ - node_folio = f2fs_get_node_folio(sbi, nid, NODE_TYPE_REGULAR); - if (IS_ERR(node_folio)) + node_entry = f2fs_get_node_cache(sbi, nid, NODE_TYPE_REGULAR); + if (IS_ERR(node_entry)) continue; /* block may become invalid during f2fs_get_node_folio */ if (check_valid_map(sbi, segno, off) == 0) { - f2fs_folio_put(node_folio, true); + f2fs_put_cache(node_entry, true); continue; } if (f2fs_get_node_info(sbi, nid, &ni, false)) { - f2fs_folio_put(node_folio, true); + f2fs_put_cache(node_entry, true); continue; } if (ni.blk_addr != start_addr + off) { - f2fs_folio_put(node_folio, true); + f2fs_put_cache(node_entry, true); continue; } - err = f2fs_move_node_folio(node_folio, gc_type); + err = f2fs_move_node_cache(node_entry, gc_type); if (!err && gc_type == FG_GC) submitted++; stat_inc_node_blk_count(sbi, 1, gc_type); @@ -1146,7 +1146,7 @@ block_t f2fs_start_bidx_of_node(unsigned int node_ofs, struct inode *inode) static bool is_alive(struct f2fs_sb_info *sbi, struct f2fs_summary *sum, struct node_info *dni, block_t blkaddr, unsigned int *nofs) { - struct folio *node_folio; + struct f2fs_cached_block *node_entry; nid_t nid; unsigned int ofs_in_node, max_addrs, base; block_t source_blkaddr; @@ -1154,12 +1154,12 @@ static bool is_alive(struct f2fs_sb_info *sbi, struct f2fs_summary *sum, nid = le32_to_cpu(sum->nid); ofs_in_node = le16_to_cpu(sum->ofs_in_node); - node_folio = f2fs_get_node_folio(sbi, nid, NODE_TYPE_REGULAR); - if (IS_ERR(node_folio)) + node_entry = f2fs_get_node_cache(sbi, nid, NODE_TYPE_REGULAR); + if (IS_ERR(node_entry)) return false; if (f2fs_get_node_info(sbi, nid, dni, false)) { - f2fs_folio_put(node_folio, true); + f2fs_put_cache(node_entry, true); return false; } @@ -1170,12 +1170,12 @@ static bool is_alive(struct f2fs_sb_info *sbi, struct f2fs_summary *sum, } if (f2fs_check_nid_range(sbi, dni->ino)) { - f2fs_folio_put(node_folio, true); + f2fs_put_cache(node_entry, true); return false; } - if (IS_INODE(node_folio)) { - base = offset_in_addr(F2FS_INODE(node_folio)); + if (IS_INODE(node_entry)) { + base = offset_in_addr(F2FS_INODE(node_entry)); max_addrs = DEF_ADDRS_PER_INODE; } else { base = 0; @@ -1185,13 +1185,13 @@ static bool is_alive(struct f2fs_sb_info *sbi, struct f2fs_summary *sum, if (base + ofs_in_node >= max_addrs) { f2fs_err(sbi, "Inconsistent blkaddr offset: base:%u, ofs_in_node:%u, max:%u, ino:%u, nid:%u", base, ofs_in_node, max_addrs, dni->ino, dni->nid); - f2fs_folio_put(node_folio, true); + f2fs_put_cache(node_entry, true); return false; } - *nofs = ofs_of_node(node_folio); - source_blkaddr = data_blkaddr(NULL, node_folio, ofs_in_node); - f2fs_folio_put(node_folio, true); + *nofs = ofs_of_node(node_entry); + source_blkaddr = data_blkaddr(NULL, node_entry, ofs_in_node); + f2fs_put_cache(node_entry, true); if (source_blkaddr != blkaddr) { #ifdef CONFIG_F2FS_CHECK_FS @@ -1283,7 +1283,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index) * don't cache encrypted data into meta inode until previous dirty * data were writebacked to avoid racing between GC and flush. */ - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); f2fs_wait_on_block_writeback(inode, dn.data_blkaddr); @@ -1394,7 +1394,7 @@ static int move_data_block(struct inode *inode, block_t bidx, * don't cache encrypted data into meta inode until previous dirty * data were writebacked to avoid racing between GC and flush. */ - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); f2fs_wait_on_block_writeback(inode, dn.data_blkaddr); @@ -1443,7 +1443,7 @@ static int move_data_block(struct inode *inode, block_t bidx, set_summary(&sum, dn.nid, dn.ofs_in_node, ni.version); /* allocate block address */ - err = f2fs_allocate_data_block(fio.sbi, NULL, fio.old_blkaddr, &newaddr, + err = f2fs_allocate_data_block(fio.sbi, fio.old_blkaddr, &newaddr, &sum, type, NULL); if (err) { f2fs_put_cache(sentry, true); @@ -1545,7 +1545,7 @@ static int move_data_page(struct inode *inode, block_t bidx, int gc_type, bool is_dirty = folio_test_dirty(folio); retry: - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); folio_mark_dirty(folio); if (folio_clear_dirty_for_io(folio)) { diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c index 2156fb1fc57d..4d27eb0b32b5 100644 --- a/fs/f2fs/inline.c +++ b/fs/f2fs/inline.c @@ -34,9 +34,9 @@ bool f2fs_may_inline_data(struct inode *inode) return !f2fs_post_read_required(inode); } -static bool inode_has_blocks(struct inode *inode, struct folio *ifolio) +static bool inode_has_blocks(struct inode *inode, struct f2fs_cached_block *ientry) { - struct f2fs_inode *ri = F2FS_INODE(ifolio); + struct f2fs_inode *ri = F2FS_INODE(ientry); int i; if (F2FS_HAS_BLOCKS(inode)) @@ -49,12 +49,12 @@ static bool inode_has_blocks(struct inode *inode, struct folio *ifolio) return false; } -bool f2fs_sanity_check_inline_data(struct inode *inode, struct folio *ifolio) +bool f2fs_sanity_check_inline_data(struct inode *inode, struct f2fs_cached_block *ientry) { if (!f2fs_has_inline_data(inode)) return false; - if (inode_has_blocks(inode, ifolio)) + if (inode_has_blocks(inode, ientry)) return false; if (!support_inline_data(inode)) @@ -80,7 +80,7 @@ bool f2fs_may_inline_dentry(struct inode *inode) return true; } -void f2fs_do_read_inline_data(struct folio *folio, struct folio *ifolio) +void f2fs_do_read_inline_data(struct folio *folio, struct f2fs_cached_block *ientry) { struct inode *inode = folio->mapping->host; @@ -92,13 +92,13 @@ void f2fs_do_read_inline_data(struct folio *folio, struct folio *ifolio) folio_zero_segment(folio, MAX_INLINE_DATA(inode), folio_size(folio)); /* Copy the whole inline data block */ - memcpy_to_folio(folio, 0, inline_data_addr(inode, ifolio), + memcpy_to_folio(folio, 0, inline_data_addr(inode, ientry), MAX_INLINE_DATA(inode)); if (!folio_test_uptodate(folio)) folio_mark_uptodate(folio); } -void f2fs_truncate_inline_inode(struct inode *inode, struct folio *ifolio, +void f2fs_truncate_inline_inode(struct inode *inode, struct f2fs_cached_block *ientry, u64 from) { void *addr; @@ -106,11 +106,11 @@ void f2fs_truncate_inline_inode(struct inode *inode, struct folio *ifolio, if (from >= MAX_INLINE_DATA(inode)) return; - addr = inline_data_addr(inode, ifolio); + addr = inline_data_addr(inode, ientry); - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_cache_wait_writeback(ientry); memset(addr + from, 0, MAX_INLINE_DATA(inode) - from); - folio_mark_dirty(ifolio); + f2fs_mark_cache_dirty(ientry); if (from == 0) clear_inode_flag(inode, FI_DATA_EXIST); @@ -118,27 +118,27 @@ void f2fs_truncate_inline_inode(struct inode *inode, struct folio *ifolio, int f2fs_read_inline_data(struct inode *inode, struct folio *folio) { - struct folio *ifolio; + struct f2fs_cached_block *ientry; - ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino); - if (IS_ERR(ifolio)) { + ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino); + if (IS_ERR(ientry)) { folio_unlock(folio); - return PTR_ERR(ifolio); + return PTR_ERR(ientry); } if (!f2fs_has_inline_data(inode)) { - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); return -EAGAIN; } if (folio->index) folio_zero_segment(folio, 0, folio_size(folio)); else - f2fs_do_read_inline_data(folio, ifolio); + f2fs_do_read_inline_data(folio, ientry); if (!folio_test_uptodate(folio)) folio_mark_uptodate(folio); - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); folio_unlock(folio); return 0; } @@ -186,7 +186,7 @@ int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio) f2fs_bug_on(F2FS_F_SB(folio), folio_test_writeback(folio)); - f2fs_do_read_inline_data(folio, dn->inode_folio); + f2fs_do_read_inline_data(folio, dn->inode_entry); folio_mark_dirty(folio); /* clear dirty state */ @@ -197,7 +197,7 @@ int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio) fio.old_blkaddr = dn->data_blkaddr; set_inode_flag(dn->inode, FI_HOT_DATA); f2fs_outplace_write_data(dn, &fio); - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); if (dirty) { inode_dec_dirty_pages(dn->inode); f2fs_remove_dirty_inode(dn->inode); @@ -207,8 +207,8 @@ int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio) set_inode_flag(dn->inode, FI_APPEND_WRITE); /* clear inline data and flag after data writeback */ - f2fs_truncate_inline_inode(dn->inode, dn->inode_folio, 0); - folio_clear_f2fs_inline(dn->inode_folio); + f2fs_truncate_inline_inode(dn->inode, dn->inode_entry, 0); + f2fs_cache_clear_inline(dn->inode_entry); clear_out: stat_dec_inline_inode(dn->inode); clear_inode_flag(dn->inode, FI_INLINE_DATA); @@ -221,7 +221,8 @@ int f2fs_convert_inline_inode(struct inode *inode) struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct dnode_of_data dn; struct f2fs_lock_context lc; - struct folio *ifolio, *folio; + struct f2fs_cached_block *ientry; + struct folio *folio; int err = 0; if (f2fs_hw_is_readonly(sbi) || f2fs_readonly(sbi->sb)) @@ -240,13 +241,13 @@ int f2fs_convert_inline_inode(struct inode *inode) f2fs_lock_op(sbi, &lc); - ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(ifolio)) { - err = PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(ientry)) { + err = PTR_ERR(ientry); goto out; } - set_new_dnode(&dn, inode, ifolio, ifolio, 0); + set_new_dnode(&dn, inode, ientry, ientry, 0); if (f2fs_has_inline_data(inode)) err = f2fs_convert_inline_folio(&dn, folio); @@ -266,31 +267,31 @@ int f2fs_convert_inline_inode(struct inode *inode) int f2fs_write_inline_data(struct inode *inode, struct folio *folio) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - struct folio *ifolio; + struct f2fs_cached_block *ientry; - ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); if (!f2fs_has_inline_data(inode)) { - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); return -EAGAIN; } f2fs_bug_on(F2FS_I_SB(inode), folio->index); - f2fs_folio_wait_writeback(ifolio, NODE, true, true); - memcpy_from_folio(inline_data_addr(inode, ifolio), + f2fs_cache_wait_writeback(ientry); + memcpy_from_folio(inline_data_addr(inode, ientry), folio, 0, MAX_INLINE_DATA(inode)); - folio_mark_dirty(ifolio); + f2fs_mark_cache_dirty(ientry); f2fs_clear_page_cache_dirty_tag(folio); set_inode_flag(inode, FI_APPEND_WRITE); set_inode_flag(inode, FI_DATA_EXIST); - folio_clear_f2fs_inline(ifolio); - f2fs_folio_put(ifolio, true); + f2fs_cache_clear_inline(ientry); + f2fs_put_cache(ientry, true); return 0; } @@ -308,40 +309,41 @@ int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entr * x o -> remove data blocks, and then recover inline_data * x x -> recover data blocks */ - if (IS_INODE(cache_folio(entry))) + if (IS_INODE(entry)) ri = &CACHED_NODE(entry)->i; if (f2fs_has_inline_data(inode) && ri && (ri->i_inline & F2FS_INLINE_DATA)) { - struct folio *ifolio; + struct f2fs_cached_block *ientry; process_inline: - ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_cache_wait_writeback(ientry); - src_addr = inline_data_addr(inode, cache_folio(entry)); - dst_addr = inline_data_addr(inode, ifolio); + src_addr = inline_data_addr(inode, entry); + dst_addr = inline_data_addr(inode, ientry); memcpy(dst_addr, src_addr, MAX_INLINE_DATA(inode)); set_inode_flag(inode, FI_INLINE_DATA); set_inode_flag(inode, FI_DATA_EXIST); - folio_mark_dirty(ifolio); - f2fs_folio_put(ifolio, true); + f2fs_mark_cache_dirty(ientry); + f2fs_put_cache(ientry, true); return 1; } if (f2fs_has_inline_data(inode)) { - struct folio *ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); - f2fs_truncate_inline_inode(inode, ifolio, 0); + struct f2fs_cached_block *ientry = f2fs_get_inode_cache(sbi, inode->i_ino); + + if (IS_ERR(ientry)) + return PTR_ERR(ientry); + f2fs_truncate_inline_inode(inode, ientry, 0); stat_dec_inline_inode(inode); clear_inode_flag(inode, FI_INLINE_DATA); - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); } else if (ri && (ri->i_inline & F2FS_INLINE_DATA)) { int ret; @@ -356,50 +358,50 @@ int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entr struct f2fs_dir_entry *f2fs_find_in_inline_dir(struct inode *dir, const struct f2fs_filename *fname, - struct folio **res_folio, + void **dentry_block, bool use_hash) { struct f2fs_sb_info *sbi = F2FS_SB(dir->i_sb); struct f2fs_dir_entry *de; struct f2fs_dentry_ptr d; - struct folio *ifolio; + struct f2fs_cached_block *ientry; void *inline_dentry; - ifolio = f2fs_get_inode_folio(sbi, dir->i_ino); - if (IS_ERR(ifolio)) { - *res_folio = ifolio; + ientry = f2fs_get_inode_cache(sbi, dir->i_ino); + if (IS_ERR(ientry)) { + *dentry_block = ientry; return NULL; } - inline_dentry = inline_data_addr(dir, ifolio); + inline_dentry = inline_data_addr(dir, ientry); make_dentry_ptr_inline(dir, &d, inline_dentry); de = f2fs_find_target_dentry(&d, fname, NULL, use_hash); - folio_unlock(ifolio); + f2fs_unlock_cache(ientry); if (IS_ERR(de)) { - *res_folio = ERR_CAST(de); + *dentry_block = ERR_CAST(de); de = NULL; } if (de) - *res_folio = ifolio; + *dentry_block = f2fs_cache_make_dentry_block(ientry); else - f2fs_folio_put(ifolio, false); + f2fs_put_cache(ientry, false); return de; } int f2fs_make_empty_inline_dir(struct inode *inode, struct inode *parent, - struct folio *ifolio) + struct f2fs_cached_block *ientry) { struct f2fs_dentry_ptr d; void *inline_dentry; - inline_dentry = inline_data_addr(inode, ifolio); + inline_dentry = inline_data_addr(inode, ientry); make_dentry_ptr_inline(inode, &d, inline_dentry); f2fs_do_make_empty_dir(inode, parent, &d); - folio_mark_dirty(ifolio); + f2fs_mark_cache_dirty(ientry); /* update i_size to MAX_INLINE_DATA */ if (i_size_read(inode) < MAX_INLINE_DATA(inode)) @@ -411,8 +413,9 @@ int f2fs_make_empty_inline_dir(struct inode *inode, struct inode *parent, * NOTE: ipage is grabbed by caller, but if any error occurs, we should * release ipage in this function. */ -static int f2fs_move_inline_dirents(struct inode *dir, struct folio *ifolio, - void *inline_dentry) +static int f2fs_move_inline_dirents(struct inode *dir, + struct f2fs_cached_block *ientry, + void *inline_dentry) { struct folio *folio; struct dnode_of_data dn; @@ -422,11 +425,11 @@ static int f2fs_move_inline_dirents(struct inode *dir, struct folio *ifolio, folio = f2fs_grab_cache_folio(dir->i_mapping, 0, true); if (IS_ERR(folio)) { - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); return PTR_ERR(folio); } - set_new_dnode(&dn, dir, ifolio, NULL, 0); + set_new_dnode(&dn, dir, ientry, NULL, 0); err = f2fs_reserve_block(&dn, 0); if (err) goto out; @@ -442,7 +445,7 @@ static int f2fs_move_inline_dirents(struct inode *dir, struct folio *ifolio, goto out; } - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); dentry_blk = folio_address(folio); @@ -465,7 +468,7 @@ static int f2fs_move_inline_dirents(struct inode *dir, struct folio *ifolio, folio_mark_dirty(folio); /* clear inline dir and flag after data writeback */ - f2fs_truncate_inline_inode(dir, ifolio, 0); + f2fs_truncate_inline_inode(dir, ientry, 0); stat_dec_inline_dir(dir); clear_inode_flag(dir, FI_INLINE_DENTRY); @@ -545,8 +548,8 @@ static int f2fs_add_inline_entries(struct inode *dir, void *inline_dentry) return err; } -static int f2fs_move_rehashed_dirents(struct inode *dir, struct folio *ifolio, - void *inline_dentry) +static int f2fs_move_rehashed_dirents(struct inode *dir, + struct f2fs_cached_block *ientry, void *inline_dentry) { void *backup_dentry; int err; @@ -554,20 +557,20 @@ static int f2fs_move_rehashed_dirents(struct inode *dir, struct folio *ifolio, backup_dentry = f2fs_kmalloc(F2FS_I_SB(dir), MAX_INLINE_DATA(dir), GFP_F2FS_ZERO); if (!backup_dentry) { - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); return -ENOMEM; } memcpy(backup_dentry, inline_dentry, MAX_INLINE_DATA(dir)); - f2fs_truncate_inline_inode(dir, ifolio, 0); + f2fs_truncate_inline_inode(dir, ientry, 0); - folio_unlock(ifolio); + f2fs_unlock_cache(ientry); err = f2fs_add_inline_entries(dir, backup_dentry); if (err) goto recover; - folio_lock(ifolio); + f2fs_lock_cache(ientry); stat_dec_inline_dir(dir); clear_inode_flag(dir, FI_INLINE_DENTRY); @@ -583,31 +586,31 @@ static int f2fs_move_rehashed_dirents(struct inode *dir, struct folio *ifolio, kfree(backup_dentry); return 0; recover: - folio_lock(ifolio); - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_lock_cache(ientry); + f2fs_cache_wait_writeback(ientry); memcpy(inline_dentry, backup_dentry, MAX_INLINE_DATA(dir)); f2fs_i_depth_write(dir, 0); f2fs_i_size_write(dir, MAX_INLINE_DATA(dir)); - folio_mark_dirty(ifolio); - f2fs_folio_put(ifolio, true); + f2fs_mark_cache_dirty(ientry); + f2fs_put_cache(ientry, true); kfree(backup_dentry); return err; } -static int do_convert_inline_dir(struct inode *dir, struct folio *ifolio, - void *inline_dentry) +static int do_convert_inline_dir(struct inode *dir, + struct f2fs_cached_block *ientry, void *inline_dentry) { if (!F2FS_I(dir)->i_dir_level) - return f2fs_move_inline_dirents(dir, ifolio, inline_dentry); + return f2fs_move_inline_dirents(dir, ientry, inline_dentry); else - return f2fs_move_rehashed_dirents(dir, ifolio, inline_dentry); + return f2fs_move_rehashed_dirents(dir, ientry, inline_dentry); } int f2fs_try_convert_inline_dir(struct inode *dir, struct dentry *dentry) { struct f2fs_sb_info *sbi = F2FS_I_SB(dir); - struct folio *ifolio; + struct f2fs_cached_block *ientry; struct f2fs_filename fname; struct f2fs_lock_context lc; void *inline_dentry = NULL; @@ -622,22 +625,22 @@ int f2fs_try_convert_inline_dir(struct inode *dir, struct dentry *dentry) if (err) goto out; - ifolio = f2fs_get_inode_folio(sbi, dir->i_ino); - if (IS_ERR(ifolio)) { - err = PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, dir->i_ino); + if (IS_ERR(ientry)) { + err = PTR_ERR(ientry); goto out_fname; } - if (f2fs_has_enough_room(dir, ifolio, &fname)) { - f2fs_folio_put(ifolio, true); + if (f2fs_has_enough_room(dir, ientry, &fname)) { + f2fs_put_cache(ientry, true); goto out_fname; } - inline_dentry = inline_data_addr(dir, ifolio); + inline_dentry = inline_data_addr(dir, ientry); - err = do_convert_inline_dir(dir, ifolio, inline_dentry); + err = do_convert_inline_dir(dir, ientry, inline_dentry); if (!err) - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); out_fname: f2fs_free_filename(&fname); out: @@ -649,24 +652,24 @@ int f2fs_add_inline_entry(struct inode *dir, const struct f2fs_filename *fname, struct inode *inode, nid_t ino, umode_t mode) { struct f2fs_sb_info *sbi = F2FS_I_SB(dir); - struct folio *ifolio; + struct f2fs_cached_block *ientry; unsigned int bit_pos; void *inline_dentry = NULL; struct f2fs_dentry_ptr d; int slots = GET_DENTRY_SLOTS(fname->disk_name.len); - struct folio *folio = NULL; + struct f2fs_cached_block *nentry = NULL; int err = 0; - ifolio = f2fs_get_inode_folio(sbi, dir->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, dir->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); - inline_dentry = inline_data_addr(dir, ifolio); + inline_dentry = inline_data_addr(dir, ientry); make_dentry_ptr_inline(dir, &d, inline_dentry); bit_pos = f2fs_room_for_filename(d.bitmap, slots, d.max); if (bit_pos >= d.max) { - err = do_convert_inline_dir(dir, ifolio, inline_dentry); + err = do_convert_inline_dir(dir, ientry, inline_dentry); if (err) return err; err = -EAGAIN; @@ -676,19 +679,19 @@ int f2fs_add_inline_entry(struct inode *dir, const struct f2fs_filename *fname, if (inode) { f2fs_down_write_nested(&F2FS_I(inode)->i_sem, SINGLE_DEPTH_NESTING); - folio = f2fs_init_inode_metadata(inode, dir, fname, ifolio); - if (IS_ERR(folio)) { - err = PTR_ERR(folio); + nentry = f2fs_init_inode_metadata(inode, dir, fname, ientry); + if (IS_ERR(nentry)) { + err = PTR_ERR(nentry); goto fail; } } - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_cache_wait_writeback(ientry); f2fs_update_dentry(ino, mode, &d, &fname->disk_name, fname->hash, bit_pos); - folio_mark_dirty(ifolio); + f2fs_mark_cache_dirty(ientry); /* we don't need to mark_inode_dirty now */ if (inode) { @@ -696,9 +699,9 @@ int f2fs_add_inline_entry(struct inode *dir, const struct f2fs_filename *fname, /* synchronize inode page's data from inode cache */ if (is_inode_flag_set(inode, FI_NEW_INODE)) - f2fs_update_inode(inode, folio); + f2fs_update_inode(inode, nentry); - f2fs_folio_put(folio, true); + f2fs_put_cache(nentry, true); } f2fs_update_parent_metadata(dir, inode, 0); @@ -706,12 +709,12 @@ int f2fs_add_inline_entry(struct inode *dir, const struct f2fs_filename *fname, if (inode) f2fs_up_write(&F2FS_I(inode)->i_sem); out: - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); return err; } void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry, - struct folio *folio, struct inode *dir, struct inode *inode) + struct f2fs_cached_block *ientry, struct inode *dir, struct inode *inode) { struct f2fs_dentry_ptr d; void *inline_dentry; @@ -719,18 +722,18 @@ void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry, unsigned int bit_pos; int i; - folio_lock(folio); - f2fs_folio_wait_writeback(folio, NODE, true, true); + f2fs_lock_cache(ientry); + f2fs_cache_wait_writeback(ientry); - inline_dentry = inline_data_addr(dir, folio); + inline_dentry = inline_data_addr(dir, ientry); make_dentry_ptr_inline(dir, &d, inline_dentry); bit_pos = dentry - d.dentry; for (i = 0; i < slots; i++) __clear_bit_le(bit_pos + i, d.bitmap); - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); + f2fs_mark_cache_dirty(ientry); + f2fs_put_cache(ientry, true); inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir)); f2fs_mark_inode_dirty_sync(dir, true); @@ -742,21 +745,21 @@ void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry, bool f2fs_empty_inline_dir(struct inode *dir) { struct f2fs_sb_info *sbi = F2FS_I_SB(dir); - struct folio *ifolio; + struct f2fs_cached_block *ientry; unsigned int bit_pos = 2; void *inline_dentry; struct f2fs_dentry_ptr d; - ifolio = f2fs_get_inode_folio(sbi, dir->i_ino); - if (IS_ERR(ifolio)) + ientry = f2fs_get_inode_cache(sbi, dir->i_ino); + if (IS_ERR(ientry)) return false; - inline_dentry = inline_data_addr(dir, ifolio); + inline_dentry = inline_data_addr(dir, ientry); make_dentry_ptr_inline(dir, &d, inline_dentry); bit_pos = find_next_bit_le(d.bitmap, d.max, bit_pos); - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); if (bit_pos < d.max) return false; @@ -768,7 +771,7 @@ int f2fs_read_inline_dir(struct file *file, struct dir_context *ctx, struct fscrypt_str *fstr) { struct inode *inode = file_inode(file); - struct folio *ifolio = NULL; + struct f2fs_cached_block *ientry = NULL; struct f2fs_dentry_ptr d; void *inline_dentry = NULL; int err; @@ -778,17 +781,17 @@ int f2fs_read_inline_dir(struct file *file, struct dir_context *ctx, if (ctx->pos == d.max) return 0; - ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); /* * f2fs_readdir was protected by inode.i_rwsem, it is safe to access * ipage without page's lock held. */ - folio_unlock(ifolio); + f2fs_unlock_cache(ientry); - inline_dentry = inline_data_addr(inode, ifolio); + inline_dentry = inline_data_addr(inode, ientry); make_dentry_ptr_inline(inode, &d, inline_dentry); @@ -796,7 +799,7 @@ int f2fs_read_inline_dir(struct file *file, struct dir_context *ctx, if (!err) ctx->pos = d.max; - f2fs_folio_put(ifolio, false); + f2fs_put_cache(ientry, false); return err < 0 ? err : 0; } @@ -807,12 +810,12 @@ int f2fs_inline_data_fiemap(struct inode *inode, __u32 flags = FIEMAP_EXTENT_DATA_INLINE | FIEMAP_EXTENT_NOT_ALIGNED | FIEMAP_EXTENT_LAST; struct node_info ni; - struct folio *ifolio; + struct f2fs_cached_block *ientry; int err = 0; - ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); if ((S_ISREG(inode->i_mode) || S_ISLNK(inode->i_mode)) && !f2fs_has_inline_data(inode)) { @@ -826,13 +829,13 @@ int f2fs_inline_data_fiemap(struct inode *inode, } if (fieinfo->fi_flags & FIEMAP_FLAG_SYNC) { - err = f2fs_write_single_node_folio(ifolio, true, false, FS_NODE_IO); + err = f2fs_write_node_cache(ientry, true, false, FS_NODE_IO); if (err) return err; - ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); + f2fs_cache_wait_writeback(ientry); } ilen = min_t(size_t, MAX_INLINE_DATA(inode), i_size_read(inode)); if (start >= ilen) @@ -847,8 +850,8 @@ int f2fs_inline_data_fiemap(struct inode *inode, if (__is_valid_data_blkaddr(ni.blk_addr)) { byteaddr = (__u64)ni.blk_addr << inode->i_sb->s_blocksize_bits; - byteaddr += (char *)inline_data_addr(inode, ifolio) - - (char *)F2FS_INODE(ifolio); + byteaddr += (char *)inline_data_addr(inode, ientry) - + (char *)F2FS_INODE(ientry); } else { f2fs_bug_on(F2FS_I_SB(inode), ni.blk_addr != NEW_ADDR); flags |= FIEMAP_EXTENT_DELALLOC | FIEMAP_EXTENT_UNKNOWN; @@ -856,6 +859,6 @@ int f2fs_inline_data_fiemap(struct inode *inode, err = fiemap_fill_next_extent(fieinfo, start, byteaddr, ilen, flags); trace_f2fs_fiemap(inode, start, byteaddr, ilen, flags, err); out: - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); return err; } diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c index c533da4d4d70..421788a9ea24 100644 --- a/fs/f2fs/inode.c +++ b/fs/f2fs/inode.c @@ -82,9 +82,9 @@ void f2fs_set_inode_flags(struct inode *inode) S_ENCRYPTED|S_VERITY|S_CASEFOLD); } -static void __get_inode_rdev(struct inode *inode, struct folio *node_folio) +static void __get_inode_rdev(struct inode *inode, struct f2fs_cached_block *ientry) { - __le32 *addr = get_dnode_addr(inode, node_folio); + __le32 *addr = get_dnode_addr(inode, ientry); if (S_ISCHR(inode->i_mode) || S_ISBLK(inode->i_mode) || S_ISFIFO(inode->i_mode) || S_ISSOCK(inode->i_mode)) { @@ -95,9 +95,9 @@ static void __get_inode_rdev(struct inode *inode, struct folio *node_folio) } } -static void __set_inode_rdev(struct inode *inode, struct folio *node_folio) +static void __set_inode_rdev(struct inode *inode, struct f2fs_cached_block *ientry) { - __le32 *addr = get_dnode_addr(inode, node_folio); + __le32 *addr = get_dnode_addr(inode, ientry); if (S_ISCHR(inode->i_mode) || S_ISBLK(inode->i_mode)) { if (old_valid_dev(inode->i_rdev)) { @@ -111,34 +111,33 @@ static void __set_inode_rdev(struct inode *inode, struct folio *node_folio) } } -static void __recover_inline_status(struct inode *inode, struct folio *ifolio) +static void __recover_inline_status(struct inode *inode, struct f2fs_cached_block *ientry) { - void *inline_data = inline_data_addr(inode, ifolio); + void *inline_data = inline_data_addr(inode, ientry); __le32 *start = inline_data; __le32 *end = start + MAX_INLINE_DATA(inode) / sizeof(__le32); while (start < end) { if (*start++) { - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_cache_wait_writeback(ientry); set_inode_flag(inode, FI_DATA_EXIST); - set_raw_inline(inode, F2FS_INODE(ifolio)); - folio_mark_dirty(ifolio); + set_raw_inline(inode, F2FS_INODE(ientry)); + f2fs_mark_cache_dirty(ientry); return; } } return; } -static -bool f2fs_enable_inode_chksum(struct f2fs_sb_info *sbi, struct folio *folio) +static bool f2fs_enable_inode_chksum(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) { - struct f2fs_inode *ri = &F2FS_NODE(folio)->i; + struct f2fs_inode *ri = F2FS_INODE(entry); if (!f2fs_sb_has_inode_chksum(sbi)) return false; - if (!IS_INODE(folio) || !(ri->i_inline & F2FS_EXTRA_ATTR)) + if (!IS_INODE(entry) || !(ri->i_inline & F2FS_EXTRA_ATTR)) return false; if (!F2FS_FITS_IN_INODE(ri, le16_to_cpu(ri->i_extra_isize), @@ -148,10 +147,10 @@ bool f2fs_enable_inode_chksum(struct f2fs_sb_info *sbi, struct folio *folio) return true; } -static __u32 f2fs_inode_chksum(struct f2fs_sb_info *sbi, struct folio *folio) +static __u32 f2fs_inode_chksum(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) { - struct f2fs_node *node = F2FS_NODE(folio); - struct f2fs_inode *ri = &node->i; + struct f2fs_node *node = CACHED_NODE(entry); + struct f2fs_inode *ri = F2FS_INODE(entry); __le32 ino = node->footer.ino; __le32 gen = ri->i_generation; __u32 chksum, chksum_seed; @@ -170,7 +169,7 @@ static __u32 f2fs_inode_chksum(struct f2fs_sb_info *sbi, struct folio *folio) return chksum; } -bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct folio *folio) +bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) { struct f2fs_inode *ri; __u32 provided, calculated; @@ -178,35 +177,33 @@ bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct folio *folio) if (unlikely(is_sbi_flag_set(sbi, SBI_IS_SHUTDOWN))) return true; -#ifdef CONFIG_F2FS_CHECK_FS - if (!f2fs_enable_inode_chksum(sbi, folio)) -#else - if (!f2fs_enable_inode_chksum(sbi, folio) || - folio_test_dirty(folio) || - folio_test_writeback(folio)) -#endif + if (!f2fs_enable_inode_chksum(sbi, entry)) return true; +#ifndef CONFIG_F2FS_CHECK_FS + if (f2fs_cache_test_dirty(entry) || f2fs_cache_test_writeback(entry)) + return true; +#endif - ri = &F2FS_NODE(folio)->i; + ri = F2FS_INODE(entry); provided = le32_to_cpu(ri->i_inode_checksum); - calculated = f2fs_inode_chksum(sbi, folio); + calculated = f2fs_inode_chksum(sbi, entry); if (provided != calculated) - f2fs_warn(sbi, "checksum invalid, nid = %lu, ino_of_node = %x, %x vs. %x", - folio->index, ino_of_node(folio), + f2fs_warn(sbi, "checksum invalid, nid = %lu, ino_of_node = %u, %x vs. %x", + entry->index, ino_of_node(entry), provided, calculated); return provided == calculated; } -void f2fs_inode_chksum_set(struct f2fs_sb_info *sbi, struct folio *folio) +void f2fs_inode_chksum_set(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) { - struct f2fs_inode *ri = &F2FS_NODE(folio)->i; + struct f2fs_inode *ri = F2FS_INODE(entry); - if (!f2fs_enable_inode_chksum(sbi, folio)) + if (!f2fs_enable_inode_chksum(sbi, entry)) return; - ri->i_inode_checksum = cpu_to_le32(f2fs_inode_chksum(sbi, folio)); + ri->i_inode_checksum = cpu_to_le32(f2fs_inode_chksum(sbi, entry)); } static bool sanity_check_compress_inode(struct inode *inode, @@ -281,28 +278,29 @@ static bool sanity_check_compress_inode(struct inode *inode, return false; } -static bool sanity_check_inode(struct inode *inode, struct folio *node_folio) +static bool sanity_check_inode(struct inode *inode, + struct f2fs_cached_block *node_entry) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct f2fs_inode_info *fi = F2FS_I(inode); - struct f2fs_inode *ri = F2FS_INODE(node_folio); + struct f2fs_inode *ri = F2FS_INODE(node_entry); unsigned long long iblocks; - iblocks = le64_to_cpu(F2FS_INODE(node_folio)->i_blocks); + iblocks = le64_to_cpu(ri->i_blocks); if (!iblocks) { f2fs_warn(sbi, "%s: corrupted inode i_blocks i_ino=%llx iblocks=%llu, run fsck to fix.", __func__, inode->i_ino, iblocks); return false; } - if (ino_of_node(node_folio) != nid_of_node(node_folio)) { + if (ino_of_node(node_entry) != nid_of_node(node_entry)) { f2fs_warn(sbi, "%s: corrupted inode footer i_ino=%llx, ino,nid: [%u, %u] run fsck to fix.", __func__, inode->i_ino, - ino_of_node(node_folio), nid_of_node(node_folio)); + ino_of_node(node_entry), nid_of_node(node_entry)); return false; } - if (ino_of_node(node_folio) == fi->i_xattr_nid) { + if (ino_of_node(node_entry) == fi->i_xattr_nid) { f2fs_warn(sbi, "%s: corrupted inode i_ino=%llx, xnid=%x, run fsck to fix.", __func__, inode->i_ino, fi->i_xattr_nid); return false; @@ -375,7 +373,7 @@ static bool sanity_check_inode(struct inode *inode, struct folio *node_folio) } } - if (f2fs_sanity_check_inline_data(inode, node_folio)) { + if (f2fs_sanity_check_inline_data(inode, node_entry)) { f2fs_warn(sbi, "%s: inode (ino=%llx, mode=%u) should not have inline_data, run fsck to fix", __func__, inode->i_ino, inode->i_mode); return false; @@ -428,7 +426,7 @@ static int do_read_inode(struct inode *inode) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct f2fs_inode_info *fi = F2FS_I(inode); - struct folio *node_folio; + struct f2fs_cached_block *node_entry; struct f2fs_inode *ri; projid_t i_projid; @@ -436,11 +434,11 @@ static int do_read_inode(struct inode *inode) if (f2fs_check_nid_range(sbi, inode->i_ino)) return -EINVAL; - node_folio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(node_folio)) - return PTR_ERR(node_folio); + node_entry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(node_entry)) + return PTR_ERR(node_entry); - ri = F2FS_INODE(node_folio); + ri = F2FS_INODE(node_entry); inode->i_mode = le16_to_cpu(ri->i_mode); i_uid_write(inode, le32_to_cpu(ri->i_uid)); @@ -490,8 +488,8 @@ static int do_read_inode(struct inode *inode) fi->i_inline_xattr_size = 0; } - if (!sanity_check_inode(inode, node_folio)) { - f2fs_folio_put(node_folio, true); + if (!sanity_check_inode(inode, node_entry)) { + f2fs_put_cache(node_entry, true); set_sbi_flag(sbi, SBI_NEED_FSCK); f2fs_handle_error(sbi, ERROR_CORRUPTED_INODE); fserror_report_file_metadata(inode, -EFSCORRUPTED, GFP_NOFS); @@ -500,17 +498,17 @@ static int do_read_inode(struct inode *inode) /* check data exist */ if (f2fs_has_inline_data(inode) && !f2fs_exist_data(inode)) - __recover_inline_status(inode, node_folio); + __recover_inline_status(inode, node_entry); /* try to recover cold bit for non-dir inode */ - if (!S_ISDIR(inode->i_mode) && !is_cold_node(node_folio)) { - f2fs_folio_wait_writeback(node_folio, NODE, true, true); - set_cold_node(node_folio, false); - folio_mark_dirty(node_folio); + if (!S_ISDIR(inode->i_mode) && !is_cold_node(node_entry)) { + f2fs_cache_wait_writeback(node_entry); + set_cold_node(node_entry, false); + f2fs_mark_cache_dirty(node_entry); } /* get rdev by using inline_info */ - __get_inode_rdev(inode, node_folio); + __get_inode_rdev(inode, node_entry); if (!f2fs_need_inode_block_update(sbi, inode->i_ino)) fi->last_disk_size = inode->i_size; @@ -553,18 +551,18 @@ static int do_read_inode(struct inode *inode) init_idisk_time(inode); - if (!sanity_check_extent_cache(inode, node_folio)) { - f2fs_folio_put(node_folio, true); + if (!sanity_check_extent_cache(inode, node_entry)) { + f2fs_put_cache(node_entry, true); f2fs_handle_error(sbi, ERROR_CORRUPTED_INODE); fserror_report_file_metadata(inode, -EFSCORRUPTED, GFP_NOFS); return -EFSCORRUPTED; } /* Need all the flag bits */ - f2fs_init_read_extent_tree(inode, node_folio); + f2fs_init_read_extent_tree(inode, node_entry); f2fs_init_age_extent_tree(inode); - f2fs_folio_put(node_folio, true); + f2fs_put_cache(node_entry, true); stat_inc_inline_xattr(inode); stat_inc_inline_inode(inode); @@ -577,8 +575,6 @@ static int do_read_inode(struct inode *inode) static bool is_meta_ino(struct f2fs_sb_info *sbi, unsigned int ino) { - if (ino == F2FS_NODE_INO(sbi)) - return true; #ifdef CONFIG_F2FS_FS_COMPRESSION if (test_opt(sbi, COMPRESS_CACHE) && ino == F2FS_COMPRESS_INO(sbi)) return true; @@ -621,10 +617,7 @@ struct inode *f2fs_iget(struct super_block *sb, unsigned long ino) make_now: f2fs_set_inode_flags(inode); - if (ino == F2FS_NODE_INO(sbi)) { - inode->i_mapping->a_ops = &f2fs_node_aops; - mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS); - } else if (ino == F2FS_COMPRESS_INO(sbi)) { + if (ino == F2FS_COMPRESS_INO(sbi)) { #ifdef CONFIG_F2FS_FS_COMPRESSION inode->i_mapping->a_ops = &f2fs_compress_aops; /* @@ -691,18 +684,19 @@ struct inode *f2fs_iget_retry(struct super_block *sb, unsigned long ino) return inode; } -void f2fs_update_inode(struct inode *inode, struct folio *node_folio) +void f2fs_update_inode(struct inode *inode, + struct f2fs_cached_block *node_entry) { struct f2fs_inode_info *fi = F2FS_I(inode); struct f2fs_inode *ri; struct extent_tree *et = fi->extent_tree[EX_READ]; - f2fs_folio_wait_writeback(node_folio, NODE, true, true); - folio_mark_dirty(node_folio); + f2fs_cache_wait_writeback(node_entry); + f2fs_mark_cache_dirty(node_entry); f2fs_inode_synced(inode); - ri = F2FS_INODE(node_folio); + ri = F2FS_INODE(node_entry); ri->i_mode = cpu_to_le16(inode->i_mode); ri->i_advise = fi->i_advise; @@ -777,27 +771,27 @@ void f2fs_update_inode(struct inode *inode, struct folio *node_folio) } } - __set_inode_rdev(inode, node_folio); + __set_inode_rdev(inode, node_entry); /* deleted inode */ if (inode->i_nlink == 0) - folio_clear_f2fs_inline(node_folio); + f2fs_cache_clear_inline(node_entry); init_idisk_time(inode); #ifdef CONFIG_F2FS_CHECK_FS - f2fs_inode_chksum_set(F2FS_I_SB(inode), node_folio); + f2fs_inode_chksum_set(F2FS_I_SB(inode), node_entry); #endif } void f2fs_update_inode_page(struct inode *inode) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - struct folio *node_folio; + struct f2fs_cached_block *node_entry; int count = 0; retry: - node_folio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(node_folio)) { - int err = PTR_ERR(node_folio); + node_entry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(node_entry)) { + int err = PTR_ERR(node_entry); /* The node block was truncated. */ if (err == -ENOENT) @@ -813,17 +807,14 @@ void f2fs_update_inode_page(struct inode *inode) f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_UPDATE_INODE); return; } - f2fs_update_inode(inode, node_folio); - f2fs_folio_put(node_folio, true); + f2fs_update_inode(inode, node_entry); + f2fs_put_cache(node_entry, true); } int f2fs_write_inode(struct inode *inode, struct writeback_control *wbc) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - if (inode->i_ino == F2FS_NODE_INO(sbi)) - return 0; - /* * atime could be updated without dirtying f2fs inode in lazytime mode */ @@ -914,8 +905,7 @@ static bool f2fs_pre_evict_inode(struct inode *inode) test_opt(sbi, COMPRESS_CACHE) && f2fs_compressed_file(inode)) f2fs_invalidate_compress_pages(sbi, inode->i_ino); - if (inode->i_ino == F2FS_NODE_INO(sbi) || - inode->i_ino == F2FS_COMPRESS_INO(sbi)) + if (inode->i_ino == F2FS_COMPRESS_INO(sbi)) return true; f2fs_bug_on(sbi, get_dirty_pages(inode)); @@ -1041,10 +1031,9 @@ static void f2fs_post_evict_inode(struct inode *inode) /* for the case f2fs_new_inode() was failed, .i_ino is zero, skip it */ if (inode->i_ino) - invalidate_mapping_pages(NODE_MAPPING(sbi), inode->i_ino, - inode->i_ino); + f2fs_invalidate_node_cache(sbi, inode->i_ino); if (xnid) - invalidate_mapping_pages(NODE_MAPPING(sbi), xnid, xnid); + f2fs_invalidate_node_cache(sbi, xnid); if (!inode->i_nlink) goto skip_record; diff --git a/fs/f2fs/namei.c b/fs/f2fs/namei.c index 37897f4321c0..43760f71e622 100644 --- a/fs/f2fs/namei.c +++ b/fs/f2fs/namei.c @@ -473,12 +473,13 @@ static int f2fs_link(struct dentry *old_dentry, struct inode *dir, struct dentry *f2fs_get_parent(struct dentry *child) { - struct folio *folio; - unsigned long ino = f2fs_inode_by_name(d_inode(child), &dotdot_name, &folio); + void *dentry_block = NULL; + unsigned long ino = f2fs_inode_by_name(d_inode(child), + &dotdot_name, &dentry_block); if (!ino) { - if (IS_ERR(folio)) - return ERR_CAST(folio); + if (IS_ERR(dentry_block)) + return ERR_CAST(dentry_block); return ERR_PTR(-ENOENT); } return d_obtain_alias(f2fs_iget(child->d_sb, ino)); @@ -489,7 +490,7 @@ static struct dentry *f2fs_lookup(struct inode *dir, struct dentry *dentry, { struct inode *inode = NULL; struct f2fs_dir_entry *de; - struct folio *folio; + void *dentry_block = NULL; struct dentry *new; nid_t ino = -1; int err = 0; @@ -507,12 +508,12 @@ static struct dentry *f2fs_lookup(struct inode *dir, struct dentry *dentry, goto out_splice; if (err) goto out; - de = __f2fs_find_entry(dir, &fname, &folio); + de = __f2fs_find_entry(dir, &fname, &dentry_block); f2fs_free_filename(&fname); if (!de) { - if (IS_ERR(folio)) { - err = PTR_ERR(folio); + if (IS_ERR(dentry_block)) { + err = PTR_ERR(dentry_block); goto out; } err = -ENOENT; @@ -520,7 +521,7 @@ static struct dentry *f2fs_lookup(struct inode *dir, struct dentry *dentry, } ino = le32_to_cpu(de->ino); - f2fs_folio_put(folio, false); + f2fs_put_dentry_block(dentry_block, false); inode = f2fs_iget(dir->i_sb, ino); if (IS_ERR(inode)) { @@ -572,7 +573,7 @@ static int f2fs_unlink(struct inode *dir, struct dentry *dentry) struct inode *inode = d_inode(dentry); struct f2fs_dir_entry *de; struct f2fs_lock_context lc; - struct folio *folio; + void *dentry_block = NULL; int err; trace_f2fs_unlink_enter(dir, dentry); @@ -592,10 +593,10 @@ static int f2fs_unlink(struct inode *dir, struct dentry *dentry) if (err) goto out; - de = f2fs_find_entry(dir, &dentry->d_name, &folio); + de = f2fs_find_entry(dir, &dentry->d_name, &dentry_block); if (!de) { - if (IS_ERR(folio)) - err = PTR_ERR(folio); + if (IS_ERR(dentry_block)) + err = PTR_ERR(dentry_block); goto out; } @@ -615,10 +616,10 @@ static int f2fs_unlink(struct inode *dir, struct dentry *dentry) err = f2fs_acquire_orphan_inode(sbi); if (err) { f2fs_unlock_op(sbi, &lc); - f2fs_folio_put(folio, false); + f2fs_put_dentry_block(dentry_block, false); goto out; } - f2fs_delete_entry(de, folio, dir, inode); + f2fs_delete_entry(de, dentry_block, dir, inode); f2fs_unlock_op(sbi, &lc); /* VFS negative dentries are incompatible with Encoding and @@ -640,7 +641,7 @@ static int f2fs_unlink(struct inode *dir, struct dentry *dentry) corrupted: err = -EFSCORRUPTED; set_sbi_flag(sbi, SBI_NEED_FSCK); - f2fs_folio_put(folio, false); + f2fs_put_dentry_block(dentry_block, false); out: trace_f2fs_unlink_exit(inode, err); return err; @@ -959,8 +960,9 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, struct inode *old_inode = d_inode(old_dentry); struct inode *new_inode = d_inode(new_dentry); struct inode *whiteout = NULL; - struct folio *old_dir_folio = NULL; - struct folio *old_folio, *new_folio = NULL; + void *old_dir_block = NULL; + void *old_block = NULL; + void *new_block = NULL; struct f2fs_dir_entry *old_dir_entry = NULL; struct f2fs_dir_entry *old_entry; struct f2fs_dir_entry *new_entry; @@ -1024,18 +1026,18 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, } err = -ENOENT; - old_entry = f2fs_find_entry(old_dir, &old_dentry->d_name, &old_folio); + old_entry = f2fs_find_entry(old_dir, &old_dentry->d_name, &old_block); if (!old_entry) { - if (IS_ERR(old_folio)) - err = PTR_ERR(old_folio); + if (IS_ERR(old_block)) + err = PTR_ERR(old_block); goto out; } if (old_is_dir && old_dir != new_dir) { - old_dir_entry = f2fs_parent_dir(old_inode, &old_dir_folio); + old_dir_entry = f2fs_parent_dir(old_inode, &old_dir_block); if (!old_dir_entry) { - if (IS_ERR(old_dir_folio)) - err = PTR_ERR(old_dir_folio); + if (IS_ERR(old_dir_block)) + err = PTR_ERR(old_dir_block); goto out_old; } } @@ -1050,10 +1052,10 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, err = -ENOENT; new_entry = f2fs_find_entry(new_dir, &new_dentry->d_name, - &new_folio); + &new_block); if (!new_entry) { - if (IS_ERR(new_folio)) - err = PTR_ERR(new_folio); + if (IS_ERR(new_block)) + err = PTR_ERR(new_block); goto out_dir; } @@ -1065,8 +1067,8 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, if (err) goto put_out_dir; - f2fs_set_link(new_dir, new_entry, new_folio, old_inode); - new_folio = NULL; + f2fs_set_link(new_dir, new_entry, new_block, old_inode); + new_block = NULL; inode_set_ctime_current(new_inode); f2fs_down_write(&F2FS_I(new_inode)->i_sem); @@ -1105,8 +1107,8 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, inode_set_ctime_current(old_inode); f2fs_mark_inode_dirty_sync(old_inode, true); - f2fs_delete_entry(old_entry, old_folio, old_dir, NULL); - old_folio = NULL; + f2fs_delete_entry(old_entry, old_block, old_dir, NULL); + old_block = NULL; if (whiteout) { set_inode_flag(whiteout, FI_INC_LINK); @@ -1124,7 +1126,7 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, } if (old_dir_entry) - f2fs_set_link(old_inode, old_dir_entry, old_dir_folio, new_dir); + f2fs_set_link(old_inode, old_dir_entry, old_dir_block, new_dir); if (old_is_dir) f2fs_i_links_write(old_dir, false); @@ -1148,12 +1150,12 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, put_out_dir: f2fs_unlock_op(sbi, &lc); - f2fs_folio_put(new_folio, false); + f2fs_put_dentry_block(new_block, false); out_dir: if (old_dir_entry) - f2fs_folio_put(old_dir_folio, false); + f2fs_put_dentry_block(old_dir_block, false); out_old: - f2fs_folio_put(old_folio, false); + f2fs_put_dentry_block(old_block, false); out: iput(whiteout); return err; @@ -1165,8 +1167,10 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, struct f2fs_sb_info *sbi = F2FS_I_SB(old_dir); struct inode *old_inode = d_inode(old_dentry); struct inode *new_inode = d_inode(new_dentry); - struct folio *old_dir_folio, *new_dir_folio; - struct folio *old_folio, *new_folio; + void *old_dir_block = NULL; + void *new_dir_block = NULL; + void *old_block = NULL; + void *new_block = NULL; struct f2fs_dir_entry *old_dir_entry = NULL, *new_dir_entry = NULL; struct f2fs_dir_entry *old_entry, *new_entry; struct f2fs_lock_context lc; @@ -1198,17 +1202,17 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, goto out; err = -ENOENT; - old_entry = f2fs_find_entry(old_dir, &old_dentry->d_name, &old_folio); + old_entry = f2fs_find_entry(old_dir, &old_dentry->d_name, &old_block); if (!old_entry) { - if (IS_ERR(old_folio)) - err = PTR_ERR(old_folio); + if (IS_ERR(old_block)) + err = PTR_ERR(old_block); goto out; } - new_entry = f2fs_find_entry(new_dir, &new_dentry->d_name, &new_folio); + new_entry = f2fs_find_entry(new_dir, &new_dentry->d_name, &new_block); if (!new_entry) { - if (IS_ERR(new_folio)) - err = PTR_ERR(new_folio); + if (IS_ERR(new_block)) + err = PTR_ERR(new_block); goto out_old; } @@ -1216,20 +1220,20 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, if (old_dir != new_dir) { if (S_ISDIR(old_inode->i_mode)) { old_dir_entry = f2fs_parent_dir(old_inode, - &old_dir_folio); + &old_dir_block); if (!old_dir_entry) { - if (IS_ERR(old_dir_folio)) - err = PTR_ERR(old_dir_folio); + if (IS_ERR(old_dir_block)) + err = PTR_ERR(old_dir_block); goto out_new; } } if (S_ISDIR(new_inode->i_mode)) { new_dir_entry = f2fs_parent_dir(new_inode, - &new_dir_folio); + &new_dir_block); if (!new_dir_entry) { - if (IS_ERR(new_dir_folio)) - err = PTR_ERR(new_dir_folio); + if (IS_ERR(new_dir_block)) + err = PTR_ERR(new_dir_block); goto out_old_dir; } } @@ -1256,14 +1260,14 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, /* update ".." directory entry info of old dentry */ if (old_dir_entry) - f2fs_set_link(old_inode, old_dir_entry, old_dir_folio, new_dir); + f2fs_set_link(old_inode, old_dir_entry, old_dir_block, new_dir); /* update ".." directory entry info of new dentry */ if (new_dir_entry) - f2fs_set_link(new_inode, new_dir_entry, new_dir_folio, old_dir); + f2fs_set_link(new_inode, new_dir_entry, new_dir_block, old_dir); /* update directory entry info of old dir inode */ - f2fs_set_link(old_dir, old_entry, old_folio, new_inode); + f2fs_set_link(old_dir, old_entry, old_block, new_inode); f2fs_down_write(&F2FS_I(old_inode)->i_sem); if (!old_dir_entry) @@ -1282,7 +1286,7 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, f2fs_mark_inode_dirty_sync(old_dir, true); /* update directory entry info of new dir inode */ - f2fs_set_link(new_dir, new_entry, new_folio, old_inode); + f2fs_set_link(new_dir, new_entry, new_block, old_inode); f2fs_down_write(&F2FS_I(new_inode)->i_sem); if (!new_dir_entry) @@ -1317,16 +1321,16 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, return 0; out_new_dir: if (new_dir_entry) { - f2fs_folio_put(new_dir_folio, false); + f2fs_put_dentry_block(new_dir_block, false); } out_old_dir: if (old_dir_entry) { - f2fs_folio_put(old_dir_folio, false); + f2fs_put_dentry_block(old_dir_block, false); } out_new: - f2fs_folio_put(new_folio, false); + f2fs_put_dentry_block(new_block, false); out_old: - f2fs_folio_put(old_folio, false); + f2fs_put_dentry_block(old_block, false); out: return err; } diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c index 398c58fc6cad..9482c86b5506 100644 --- a/fs/f2fs/node.c +++ b/fs/f2fs/node.c @@ -13,6 +13,7 @@ #include <linux/folio_batch.h> #include <linux/swap.h> #include <linux/fserror.h> +#include <linux/freezer.h> #include "f2fs.h" #include "node.h" @@ -130,16 +131,6 @@ bool f2fs_available_free_memory(struct f2fs_sb_info *sbi, int type) return res; } -static void clear_node_folio_dirty(struct folio *folio) -{ - if (folio_test_dirty(folio)) { - f2fs_clear_page_cache_dirty_tag(folio); - folio_clear_dirty_for_io(folio); - dec_page_count(F2FS_F_SB(folio), F2FS_DIRTY_NODES); - } - folio_clear_uptodate(folio); -} - static struct f2fs_cached_block *get_current_nat_cache(struct f2fs_sb_info *sbi, nid_t nid) { @@ -328,9 +319,12 @@ static unsigned int __gang_lookup_nat_set(struct f2fs_nm_info *nm_i, start, nr); } -bool f2fs_in_warm_node_list(struct folio *folio) + + +bool f2fs_in_warm_node_list(struct f2fs_cached_block *entry) { - return is_node_folio(folio) && IS_DNODE(folio) && is_cold_node(folio); + return f2fs_is_node_cache(entry) && IS_DNODE(entry) && + is_cold_node(entry); } void f2fs_init_fsync_node_info(struct f2fs_sb_info *sbi) @@ -342,7 +336,7 @@ void f2fs_init_fsync_node_info(struct f2fs_sb_info *sbi) } static unsigned int f2fs_add_fsync_node_entry(struct f2fs_sb_info *sbi, - struct folio *folio) + struct f2fs_cached_block *entry) { struct fsync_node_entry *fn; unsigned long flags; @@ -351,8 +345,8 @@ static unsigned int f2fs_add_fsync_node_entry(struct f2fs_sb_info *sbi, fn = f2fs_kmem_cache_alloc(fsync_node_entry_slab, GFP_NOFS, true, NULL); - folio_get(folio); - fn->folio = folio; + f2fs_cache_get(entry); + fn->entry = entry; INIT_LIST_HEAD(&fn->list); spin_lock_irqsave(&sbi->fsync_node_lock, flags); @@ -365,19 +359,19 @@ static unsigned int f2fs_add_fsync_node_entry(struct f2fs_sb_info *sbi, return seq_id; } -void f2fs_del_fsync_node_entry(struct f2fs_sb_info *sbi, struct folio *folio) +void f2fs_del_fsync_node_entry(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) { struct fsync_node_entry *fn; unsigned long flags; spin_lock_irqsave(&sbi->fsync_node_lock, flags); list_for_each_entry(fn, &sbi->fsync_node_list, list) { - if (fn->folio == folio) { + if (fn->entry == entry) { list_del(&fn->list); sbi->fsync_node_num--; spin_unlock_irqrestore(&sbi->fsync_node_lock, flags); + f2fs_put_cache(fn->entry, false); kmem_cache_free(fsync_node_entry_slab, fn); - folio_put(folio); return; } } @@ -670,9 +664,9 @@ int f2fs_get_node_info(struct f2fs_sb_info *sbi, nid_t nid, /* * readahead MAX_RA_NODE number of node pages. */ -static void f2fs_ra_node_pages(struct folio *parent, int start, int n) +static void f2fs_ra_node_pages(struct f2fs_cached_block *parent, int start, int n) { - struct f2fs_sb_info *sbi = F2FS_F_SB(parent); + struct f2fs_sb_info *sbi = parent->cache->sbi; struct blk_plug plug; int i, end; nid_t nid; @@ -801,7 +795,8 @@ static int get_node_path(struct inode *inode, long block, return level; } -static struct folio *f2fs_get_node_folio_ra(struct folio *parent, int start); +static struct f2fs_cached_block *f2fs_get_node_cache_ra( + struct f2fs_cached_block *parent, int start); /* * Caller should call f2fs_put_dnode(dn). @@ -811,8 +806,8 @@ static struct folio *f2fs_get_node_folio_ra(struct folio *parent, int start); int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode) { struct f2fs_sb_info *sbi = F2FS_I_SB(dn->inode); - struct folio *nfolio[4]; - struct folio *parent = NULL; + struct f2fs_cached_block *nentry[4]; + struct f2fs_cached_block *parent = NULL; int offset[4]; unsigned int noffset[4]; nid_t nids[4]; @@ -825,26 +820,26 @@ int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode) nids[0] = dn->inode->i_ino; - if (!dn->inode_folio) { - nfolio[0] = f2fs_get_inode_folio(sbi, nids[0]); - if (IS_ERR(nfolio[0])) - return PTR_ERR(nfolio[0]); + if (!dn->inode_entry) { + nentry[0] = f2fs_get_inode_cache(sbi, nids[0]); + if (IS_ERR(nentry[0])) + return PTR_ERR(nentry[0]); } else { - nfolio[0] = dn->inode_folio; + nentry[0] = dn->inode_entry; } /* if inline_data is set, should not report any block indices */ if (f2fs_has_inline_data(dn->inode) && index) { err = -ENOENT; - f2fs_folio_put(nfolio[0], true); + f2fs_put_cache(nentry[0], true); goto release_out; } - parent = nfolio[0]; + parent = nentry[0]; if (level != 0) nids[1] = get_nid(parent, offset[0], true); - dn->inode_folio = nfolio[0]; - dn->inode_folio_locked = true; + dn->inode_entry = nentry[0]; + dn->inode_entry_locked = true; /* get indirect or direct nodes */ for (i = 1; i <= level; i++) { @@ -868,10 +863,10 @@ int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode) } dn->nid = nids[i]; - nfolio[i] = f2fs_new_node_folio(dn, noffset[i]); - if (IS_ERR(nfolio[i])) { + nentry[i] = f2fs_new_node_cache(dn, noffset[i]); + if (IS_ERR(nentry[i])) { f2fs_alloc_nid_failed(sbi, nids[i]); - err = PTR_ERR(nfolio[i]); + err = PTR_ERR(nentry[i]); goto release_pages; } @@ -879,37 +874,37 @@ int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode) f2fs_alloc_nid_done(sbi, nids[i]); done = true; } else if (mode == LOOKUP_NODE_RA && i == level && level > 1) { - nfolio[i] = f2fs_get_node_folio_ra(parent, offset[i - 1]); - if (IS_ERR(nfolio[i])) { - err = PTR_ERR(nfolio[i]); + nentry[i] = f2fs_get_node_cache_ra(parent, offset[i - 1]); + if (IS_ERR(nentry[i])) { + err = PTR_ERR(nentry[i]); goto release_pages; } done = true; } if (i == 1) { - dn->inode_folio_locked = false; - folio_unlock(parent); + dn->inode_entry_locked = false; + f2fs_unlock_cache(parent); } else { - f2fs_folio_put(parent, true); + f2fs_put_cache(parent, true); } if (!done) { - nfolio[i] = f2fs_get_node_folio(sbi, nids[i], + nentry[i] = f2fs_get_node_cache(sbi, nids[i], NODE_TYPE_NON_INODE); - if (IS_ERR(nfolio[i])) { - err = PTR_ERR(nfolio[i]); - f2fs_folio_put(nfolio[0], false); + if (IS_ERR(nentry[i])) { + err = PTR_ERR(nentry[i]); + f2fs_put_cache(nentry[0], false); goto release_out; } } if (i < level) { - parent = nfolio[i]; + parent = nentry[i]; nids[i + 1] = get_nid(parent, offset[i], false); } } dn->nid = nids[level]; dn->ofs_in_node = offset[level]; - dn->node_folio = nfolio[level]; + dn->node_entry = nentry[level]; dn->data_blkaddr = f2fs_data_blkaddr(dn); if (is_inode_flag_set(dn->inode, FI_COMPRESSED_FILE) && @@ -930,9 +925,9 @@ int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode) if (!c_len) goto out; - blkaddr = data_blkaddr(dn->inode, dn->node_folio, ofs_in_node); + blkaddr = data_blkaddr(dn->inode, dn->node_entry, ofs_in_node); if (blkaddr == COMPRESS_ADDR) - blkaddr = data_blkaddr(dn->inode, dn->node_folio, + blkaddr = data_blkaddr(dn->inode, dn->node_entry, ofs_in_node + 1); f2fs_update_read_extent_tree_range_compressed(dn->inode, @@ -942,12 +937,12 @@ int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode) return 0; release_pages: - f2fs_folio_put(parent, true); + f2fs_put_cache(parent, true); if (i > 1) - f2fs_folio_put(nfolio[0], false); + f2fs_put_cache(nentry[0], false); release_out: - dn->inode_folio = NULL; - dn->node_folio = NULL; + dn->inode_entry = NULL; + dn->node_entry = NULL; if (err == -ENOENT) { dn->cur_level = i; dn->max_level = level; @@ -988,16 +983,15 @@ static int truncate_node(struct dnode_of_data *dn) f2fs_inode_synced(dn->inode); } - clear_node_folio_dirty(dn->node_folio); + f2fs_drop_cache_dirty(dn->node_entry); set_sbi_flag(sbi, SBI_IS_DIRTY); - index = dn->node_folio->index; - f2fs_folio_put(dn->node_folio, true); + index = dn->node_entry->index; + f2fs_put_cache(dn->node_entry, true); - invalidate_mapping_pages(NODE_MAPPING(sbi), - index, index); + f2fs_invalidate_node_cache(sbi, index); - dn->node_folio = NULL; + dn->node_entry = NULL; trace_f2fs_truncate_node(dn->inode, dn->nid, ni.blk_addr); return 0; @@ -1006,35 +1000,35 @@ static int truncate_node(struct dnode_of_data *dn) static int truncate_dnode(struct dnode_of_data *dn) { struct f2fs_sb_info *sbi = F2FS_I_SB(dn->inode); - struct folio *folio; + struct f2fs_cached_block *entry; int err; if (dn->nid == 0) return 1; /* get direct node */ - folio = f2fs_get_node_folio(sbi, dn->nid, NODE_TYPE_NON_INODE); - if (PTR_ERR(folio) == -ENOENT) + entry = f2fs_get_node_cache(sbi, dn->nid, NODE_TYPE_NON_INODE); + if (PTR_ERR(entry) == -ENOENT) return 1; - else if (IS_ERR(folio)) - return PTR_ERR(folio); + else if (IS_ERR(entry)) + return PTR_ERR(entry); - if (IS_INODE(folio) || ino_of_node(folio) != dn->inode->i_ino) { + if (IS_INODE(entry) || ino_of_node(entry) != dn->inode->i_ino) { f2fs_err(sbi, "incorrect node reference, ino: %llu, nid: %u, ino_of_node: %u", - dn->inode->i_ino, dn->nid, ino_of_node(folio)); + dn->inode->i_ino, dn->nid, ino_of_node(entry)); set_sbi_flag(sbi, SBI_NEED_FSCK); f2fs_handle_error(sbi, ERROR_INVALID_NODE_REFERENCE); - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); return -EFSCORRUPTED; } /* Make dnode_of_data for parameter */ - dn->node_folio = folio; + dn->node_entry = entry; dn->ofs_in_node = 0; f2fs_truncate_data_blocks_range(dn, ADDRS_PER_BLOCK(dn->inode)); err = truncate_node(dn); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); return err; } @@ -1045,7 +1039,7 @@ static int truncate_nodes(struct dnode_of_data *dn, unsigned int nofs, int ofs, int depth) { struct dnode_of_data rdn = *dn; - struct folio *folio; + struct f2fs_cached_block *entry; struct f2fs_node *rn; nid_t child_nid; unsigned int child_nofs; @@ -1057,16 +1051,16 @@ static int truncate_nodes(struct dnode_of_data *dn, unsigned int nofs, trace_f2fs_truncate_nodes_enter(dn->inode, dn->nid, dn->data_blkaddr); - folio = f2fs_get_node_folio(F2FS_I_SB(dn->inode), dn->nid, + entry = f2fs_get_node_cache(F2FS_I_SB(dn->inode), dn->nid, NODE_TYPE_NON_INODE); - if (IS_ERR(folio)) { - trace_f2fs_truncate_nodes_exit(dn->inode, PTR_ERR(folio)); - return PTR_ERR(folio); + if (IS_ERR(entry)) { + trace_f2fs_truncate_nodes_exit(dn->inode, PTR_ERR(entry)); + return PTR_ERR(entry); } - f2fs_ra_node_pages(folio, ofs, NIDS_PER_BLOCK); + f2fs_ra_node_pages(entry, ofs, NIDS_PER_BLOCK); - rn = F2FS_NODE(folio); + rn = CACHED_NODE(entry); if (depth < 3) { for (i = ofs; i < NIDS_PER_BLOCK; i++, freed++) { child_nid = le32_to_cpu(rn->in.nid[i]); @@ -1076,7 +1070,7 @@ static int truncate_nodes(struct dnode_of_data *dn, unsigned int nofs, ret = truncate_dnode(&rdn); if (ret < 0) goto out_err; - if (set_nid(folio, i, 0, false)) + if (set_nid(entry, i, 0, false)) dn->node_changed = true; } } else { @@ -1090,7 +1084,7 @@ static int truncate_nodes(struct dnode_of_data *dn, unsigned int nofs, rdn.nid = child_nid; ret = truncate_nodes(&rdn, child_nofs, 0, depth - 1); if (ret == (NIDS_PER_BLOCK + 1)) { - if (set_nid(folio, i, 0, false)) + if (set_nid(entry, i, 0, false)) dn->node_changed = true; child_nofs += ret; } else if (ret < 0 && ret != -ENOENT) { @@ -1102,19 +1096,19 @@ static int truncate_nodes(struct dnode_of_data *dn, unsigned int nofs, if (!ofs) { /* remove current indirect node */ - dn->node_folio = folio; + dn->node_entry = entry; ret = truncate_node(dn); if (ret) goto out_err; freed++; } else { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); } trace_f2fs_truncate_nodes_exit(dn->inode, freed); return freed; out_err: - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); trace_f2fs_truncate_nodes_exit(dn->inode, ret); return ret; } @@ -1122,60 +1116,60 @@ static int truncate_nodes(struct dnode_of_data *dn, unsigned int nofs, static int truncate_partial_nodes(struct dnode_of_data *dn, int *offset, int depth) { - struct folio *folios[2]; + struct f2fs_cached_block *entries[2]; nid_t nid[3]; nid_t child_nid; int err = 0; int i; int idx = depth - 2; - nid[0] = get_nid(dn->inode_folio, offset[0], true); + nid[0] = get_nid(dn->inode_entry, offset[0], true); if (!nid[0]) return 0; /* get indirect nodes in the path */ for (i = 0; i < idx + 1; i++) { /* reference count'll be increased */ - folios[i] = f2fs_get_node_folio(F2FS_I_SB(dn->inode), nid[i], + entries[i] = f2fs_get_node_cache(F2FS_I_SB(dn->inode), nid[i], NODE_TYPE_NON_INODE); - if (IS_ERR(folios[i])) { - err = PTR_ERR(folios[i]); + if (IS_ERR(entries[i])) { + err = PTR_ERR(entries[i]); idx = i - 1; goto fail; } - nid[i + 1] = get_nid(folios[i], offset[i + 1], false); + nid[i + 1] = get_nid(entries[i], offset[i + 1], false); } - f2fs_ra_node_pages(folios[idx], offset[idx + 1], NIDS_PER_BLOCK); + f2fs_ra_node_pages(entries[idx], offset[idx + 1], NIDS_PER_BLOCK); /* free direct nodes linked to a partial indirect node */ for (i = offset[idx + 1]; i < NIDS_PER_BLOCK; i++) { - child_nid = get_nid(folios[idx], i, false); + child_nid = get_nid(entries[idx], i, false); if (!child_nid) continue; dn->nid = child_nid; err = truncate_dnode(dn); if (err < 0) goto fail; - if (set_nid(folios[idx], i, 0, false)) + if (set_nid(entries[idx], i, 0, false)) dn->node_changed = true; } if (offset[idx + 1] == 0) { - dn->node_folio = folios[idx]; + dn->node_entry = entries[idx]; dn->nid = nid[idx]; err = truncate_node(dn); if (err) goto fail; } else { - f2fs_folio_put(folios[idx], true); + f2fs_put_cache(entries[idx], true); } offset[idx]++; offset[idx + 1] = 0; idx--; fail: for (i = idx; i >= 0; i--) - f2fs_folio_put(folios[i], true); + f2fs_put_cache(entries[i], true); trace_f2fs_truncate_partial_nodes(dn->inode, nid, depth, err); @@ -1192,7 +1186,7 @@ int f2fs_truncate_inode_blocks(struct inode *inode, pgoff_t from) int level, offset[4], noffset[4]; unsigned int nofs = 0; struct dnode_of_data dn; - struct folio *folio; + struct f2fs_cached_block *entry; trace_f2fs_truncate_inode_blocks_enter(inode, from); @@ -1209,14 +1203,14 @@ int f2fs_truncate_inode_blocks(struct inode *inode, pgoff_t from) return level; } - folio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(folio)) { - trace_f2fs_truncate_inode_blocks_exit(inode, PTR_ERR(folio)); - return PTR_ERR(folio); + entry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(entry)) { + trace_f2fs_truncate_inode_blocks_exit(inode, PTR_ERR(entry)); + return PTR_ERR(entry); } - set_new_dnode(&dn, inode, folio, NULL, 0); - folio_unlock(folio); + set_new_dnode(&dn, inode, entry, NULL, 0); + f2fs_unlock_cache(entry); switch (level) { case 0: @@ -1246,7 +1240,7 @@ int f2fs_truncate_inode_blocks(struct inode *inode, pgoff_t from) skip_partial: while (cont) { - dn.nid = get_nid(folio, offset[0], true); + dn.nid = get_nid(entry, offset[0], true); switch (offset[0]) { case NODE_DIR1_BLOCK: case NODE_DIR2_BLOCK: @@ -1267,7 +1261,7 @@ int f2fs_truncate_inode_blocks(struct inode *inode, pgoff_t from) BUG(); } if (err == -ENOENT) { - set_sbi_flag(F2FS_F_SB(folio), SBI_NEED_FSCK); + set_sbi_flag(sbi, SBI_NEED_FSCK); f2fs_handle_error(sbi, ERROR_INVALID_BLKADDR); fserror_report_file_metadata(dn.inode, -EFSCORRUPTED, GFP_NOFS); @@ -1280,18 +1274,18 @@ int f2fs_truncate_inode_blocks(struct inode *inode, pgoff_t from) } if (err < 0) goto fail; - if (offset[1] == 0 && get_nid(folio, offset[0], true)) { - folio_lock(folio); - BUG_ON(!is_node_folio(folio)); - set_nid(folio, offset[0], 0, true); - folio_unlock(folio); + if (offset[1] == 0 && get_nid(entry, offset[0], true)) { + f2fs_lock_cache(entry); + f2fs_bug_on(sbi, !f2fs_is_node_cache(entry)); + set_nid(entry, offset[0], 0, true); + f2fs_unlock_cache(entry); } offset[1] = 0; offset[0]++; nofs += err; } fail: - f2fs_folio_put(folio, false); + f2fs_put_cache(entry, false); trace_f2fs_truncate_inode_blocks_exit(inode, err); return err > 0 ? 0 : err; } @@ -1302,20 +1296,20 @@ int f2fs_truncate_xattr_node(struct inode *inode) struct f2fs_sb_info *sbi = F2FS_I_SB(inode); nid_t nid = F2FS_I(inode)->i_xattr_nid; struct dnode_of_data dn; - struct folio *nfolio; + struct f2fs_cached_block *nentry; int err; if (!nid) return 0; - nfolio = f2fs_get_xnode_folio(sbi, nid); - if (IS_ERR(nfolio)) - return PTR_ERR(nfolio); + nentry = f2fs_get_xnode_cache(sbi, nid); + if (IS_ERR(nentry)) + return PTR_ERR(nentry); - set_new_dnode(&dn, inode, NULL, nfolio, nid); + set_new_dnode(&dn, inode, NULL, nentry, nid); err = truncate_node(&dn); if (err) { - f2fs_folio_put(nfolio, true); + f2fs_put_cache(nentry, true); return err; } @@ -1372,30 +1366,30 @@ int f2fs_remove_inode_page(struct inode *inode) return 0; } -struct folio *f2fs_new_inode_folio(struct inode *inode) +struct f2fs_cached_block *f2fs_new_inode_cache(struct inode *inode) { struct dnode_of_data dn; /* allocate inode page for new inode */ set_new_dnode(&dn, inode, NULL, NULL, inode->i_ino); - /* caller should f2fs_folio_put(folio, true); */ - return f2fs_new_node_folio(&dn, 0); + /* caller should f2fs_put_cache(entry, true); */ + return f2fs_new_node_cache(&dn, 0); } -struct folio *f2fs_new_node_folio(struct dnode_of_data *dn, unsigned int ofs) +struct f2fs_cached_block *f2fs_new_node_cache(struct dnode_of_data *dn, unsigned int ofs) { struct f2fs_sb_info *sbi = F2FS_I_SB(dn->inode); struct node_info new_ni; - struct folio *folio; + struct f2fs_cached_block *entry; int err; if (unlikely(is_inode_flag_set(dn->inode, FI_NO_ALLOC))) return ERR_PTR(-EPERM); - folio = f2fs_grab_cache_folio(NODE_MAPPING(sbi), dn->nid, false); - if (IS_ERR(folio)) - return folio; + entry = f2fs_grab_node_cache(sbi, dn->nid); + if (IS_ERR(entry)) + return entry; if (unlikely((err = inc_valid_node_count(sbi, dn->inode, !ofs)))) goto fail; @@ -1411,7 +1405,7 @@ struct folio *f2fs_new_node_folio(struct dnode_of_data *dn, unsigned int ofs) dec_valid_node_count(sbi, dn->inode, !ofs); set_sbi_flag(sbi, SBI_NEED_FSCK); f2fs_warn_ratelimited(sbi, - "f2fs_new_node_folio: inconsistent nat entry, " + "f2fs_new_node_cache: inconsistent nat entry, " "ino:%u, nid:%u, blkaddr:%u, ver:%u, flag:%u", new_ni.ino, new_ni.nid, new_ni.blk_addr, new_ni.version, new_ni.flag); @@ -1426,12 +1420,11 @@ struct folio *f2fs_new_node_folio(struct dnode_of_data *dn, unsigned int ofs) new_ni.version = 0; set_node_addr(sbi, &new_ni, NEW_ADDR, false); - f2fs_folio_wait_writeback(folio, NODE, true, true); - fill_node_footer(folio, dn->nid, dn->inode->i_ino, ofs, true); - set_cold_node(folio, S_ISDIR(dn->inode->i_mode)); - if (!folio_test_uptodate(folio)) - folio_mark_uptodate(folio); - if (folio_mark_dirty(folio)) + f2fs_cache_wait_writeback(entry); + fill_node_footer(entry, dn->nid, dn->inode->i_ino, ofs, true); + set_cold_node(entry, S_ISDIR(dn->inode->i_mode)); + f2fs_cache_set_uptodate(entry); + if (f2fs_mark_cache_dirty(entry)) dn->node_changed = true; if (f2fs_has_xattr_block(ofs)) @@ -1439,54 +1432,54 @@ struct folio *f2fs_new_node_folio(struct dnode_of_data *dn, unsigned int ofs) if (ofs == 0) inc_valid_inode_count(sbi); - return folio; + return entry; fail: - clear_node_folio_dirty(folio); - f2fs_folio_put(folio, true); + f2fs_drop_cache_dirty(entry); + f2fs_put_cache(entry, true); return ERR_PTR(err); } /* * Caller should do after getting the following values. - * 0: f2fs_folio_put(folio, false) - * LOCKED_PAGE or error: f2fs_folio_put(folio, true) + * 0: f2fs_put_cache(cache, false) + * LOCKED_PAGE or error: f2fs_put_cache(entry, true) */ -static int read_node_folio(struct folio *folio, blk_opf_t op_flags) +static int read_node_cache(struct f2fs_cached_block *entry, blk_opf_t op_flags) { - struct f2fs_sb_info *sbi = F2FS_F_SB(folio); + struct f2fs_sb_info *sbi = entry->cache->sbi; struct node_info ni; struct f2fs_io_info fio = { .sbi = sbi, .type = NODE, .op = REQ_OP_READ, .op_flags = op_flags, - .folio = folio, .encrypted_page = NULL, + .cache_entry = entry, + .is_cache = 1, }; int err; - if (folio_test_uptodate(folio)) { - if (!f2fs_inode_chksum_verify(sbi, folio)) { - folio_clear_uptodate(folio); + if (f2fs_cache_test_uptodate(entry)) { + if (!f2fs_inode_chksum_verify(sbi, entry)) { + f2fs_cache_clear_uptodate(entry); return -EFSBADCRC; } return LOCKED_PAGE; } - err = f2fs_get_node_info(sbi, folio->index, &ni, false); + err = f2fs_get_node_info(sbi, entry->index, &ni, false); if (err) return err; /* NEW_ADDR can be seen, after cp_error drops some dirty node pages */ if (unlikely(ni.blk_addr == NULL_ADDR || ni.blk_addr == NEW_ADDR)) { - folio_clear_uptodate(folio); + f2fs_cache_clear_uptodate(entry); return -ENOENT; } fio.new_blkaddr = fio.old_blkaddr = ni.blk_addr; - err = f2fs_submit_page_bio(&fio); - + err = f2fs_submit_cache_read(&fio); if (!err) f2fs_update_iostat(sbi, NULL, FS_NODE_READ_IO, F2FS_BLKSIZE); @@ -1498,7 +1491,7 @@ static int read_node_folio(struct folio *folio, blk_opf_t op_flags) */ void f2fs_ra_node_page(struct f2fs_sb_info *sbi, nid_t nid) { - struct folio *afolio; + struct f2fs_cached_block *entry; int err; if (!nid) @@ -1506,29 +1499,29 @@ void f2fs_ra_node_page(struct f2fs_sb_info *sbi, nid_t nid) if (f2fs_check_nid_range(sbi, nid)) return; - afolio = xa_load(&NODE_MAPPING(sbi)->i_pages, nid); - if (afolio) + entry = xa_load(&META_CACHE(sbi)->root, nid); + if (entry) return; - afolio = f2fs_grab_cache_folio(NODE_MAPPING(sbi), nid, false); - if (IS_ERR(afolio)) + entry = f2fs_grab_node_cache(sbi, nid); + if (IS_ERR(entry)) return; - err = read_node_folio(afolio, REQ_RAHEAD); - f2fs_folio_put(afolio, err ? true : false); + err = read_node_cache(entry, REQ_RAHEAD); + f2fs_put_cache(entry, err ? true : false); } int f2fs_sanity_check_node_footer(struct f2fs_sb_info *sbi, - struct folio *folio, pgoff_t nid, + struct f2fs_cached_block *entry, pgoff_t nid, enum node_type ntype, bool in_irq) { bool is_inode, is_xnode; - if (unlikely(nid != nid_of_node(folio))) + if (unlikely(nid != nid_of_node(entry))) goto out_err; - is_inode = IS_INODE(folio); - is_xnode = f2fs_has_xattr_block(ofs_of_node(folio)); + is_inode = IS_INODE(entry); + is_xnode = f2fs_has_xattr_block(ofs_of_node(entry)); switch (ntype) { case NODE_TYPE_REGULAR: @@ -1561,20 +1554,20 @@ int f2fs_sanity_check_node_footer(struct f2fs_sb_info *sbi, set_sbi_flag(sbi, SBI_NEED_FSCK); f2fs_warn_ratelimited(sbi, "inconsistent node block, node_type:%d, nid:%lu, " "node_footer[nid:%u,ino:%u,ofs:%u,cpver:%llu,blkaddr:%u]", - ntype, nid, nid_of_node(folio), ino_of_node(folio), - ofs_of_node(folio), cpver_of_node(folio), - next_blkaddr_of_node(folio)); + ntype, nid, nid_of_node(entry), ino_of_node(entry), + ofs_of_node(entry), cpver_of_node(entry), + next_blkaddr_of_node(entry)); f2fs_handle_error(sbi, ERROR_INCONSISTENT_FOOTER); - fserror_report_file_metadata(folio->mapping->host, - -EFSCORRUPTED, in_irq ? GFP_NOWAIT : GFP_NOFS); + fserror_report_metadata(sbi->sb, -EFSCORRUPTED, + in_irq ? GFP_NOWAIT : GFP_NOFS); return -EFSCORRUPTED; } -static struct folio *__get_node_folio(struct f2fs_sb_info *sbi, pgoff_t nid, - struct folio *parent, int start, enum node_type ntype) +static struct f2fs_cached_block *__get_node_cache(struct f2fs_sb_info *sbi, pgoff_t nid, + struct f2fs_cached_block *parent, int start, enum node_type ntype) { - struct folio *folio; + struct f2fs_cached_block *entry; int err; if (!nid) @@ -1582,71 +1575,71 @@ static struct folio *__get_node_folio(struct f2fs_sb_info *sbi, pgoff_t nid, if (f2fs_check_nid_range(sbi, nid)) return ERR_PTR(-EINVAL); repeat: - folio = f2fs_grab_cache_folio(NODE_MAPPING(sbi), nid, false); - if (IS_ERR(folio)) - return folio; + entry = f2fs_grab_node_cache(sbi, nid); + if (IS_ERR(entry)) + return entry; - err = read_node_folio(folio, 0); + err = read_node_cache(entry, 0); if (err < 0) goto out_put_err; if (err == LOCKED_PAGE) - goto page_hit; + goto entry_hit; if (parent) f2fs_ra_node_pages(parent, start + 1, MAX_RA_NODE); - folio_lock(folio); + f2fs_lock_cache(entry); - if (unlikely(!is_node_folio(folio))) { - f2fs_folio_put(folio, true); + if (unlikely(!f2fs_is_node_cache(entry))) { + f2fs_put_cache(entry, true); goto repeat; } - if (unlikely(!folio_test_uptodate(folio))) { + if (unlikely(!f2fs_cache_test_uptodate(entry))) { err = -EIO; goto out_put_err; } - if (!f2fs_inode_chksum_verify(sbi, folio)) { + if (!f2fs_inode_chksum_verify(sbi, entry)) { err = -EFSBADCRC; goto out_err; } -page_hit: - err = f2fs_sanity_check_node_footer(sbi, folio, nid, ntype, false); +entry_hit: + err = f2fs_sanity_check_node_footer(sbi, entry, nid, ntype, false); if (!err) - return folio; + return entry; out_err: - clear_node_folio_dirty(folio); + f2fs_clear_cache_dirty(entry); out_put_err: /* ENOENT comes from read_node_folio which is not an error. */ if (err != -ENOENT) - f2fs_handle_page_eio(sbi, folio->index, NODE); - f2fs_folio_put(folio, true); + f2fs_handle_page_eio(sbi, entry->index, NODE); + f2fs_put_cache(entry, true); return ERR_PTR(err); } -struct folio *f2fs_get_node_folio(struct f2fs_sb_info *sbi, pgoff_t nid, +struct f2fs_cached_block *f2fs_get_node_cache(struct f2fs_sb_info *sbi, pgoff_t nid, enum node_type node_type) { - return __get_node_folio(sbi, nid, NULL, 0, node_type); + return __get_node_cache(sbi, nid, NULL, 0, node_type); } -struct folio *f2fs_get_inode_folio(struct f2fs_sb_info *sbi, pgoff_t ino) +struct f2fs_cached_block *f2fs_get_inode_cache(struct f2fs_sb_info *sbi, pgoff_t ino) { - return __get_node_folio(sbi, ino, NULL, 0, NODE_TYPE_INODE); + return __get_node_cache(sbi, ino, NULL, 0, NODE_TYPE_INODE); } -struct folio *f2fs_get_xnode_folio(struct f2fs_sb_info *sbi, pgoff_t xnid) +struct f2fs_cached_block *f2fs_get_xnode_cache(struct f2fs_sb_info *sbi, pgoff_t xnid) { - return __get_node_folio(sbi, xnid, NULL, 0, NODE_TYPE_XATTR); + return __get_node_cache(sbi, xnid, NULL, 0, NODE_TYPE_XATTR); } -static struct folio *f2fs_get_node_folio_ra(struct folio *parent, int start) +static struct f2fs_cached_block *f2fs_get_node_cache_ra(struct f2fs_cached_block *parent, int start) { - struct f2fs_sb_info *sbi = F2FS_F_SB(parent); + struct f2fs_sb_info *sbi = parent->cache->sbi; nid_t nid = get_nid(parent, start, false); - return __get_node_folio(sbi, nid, parent, start, NODE_TYPE_NON_IXNODE); + return __get_node_cache(sbi, nid, parent, start, NODE_TYPE_NON_IXNODE); } static void flush_inline_data(struct f2fs_sb_info *sbi, nid_t ino) @@ -1685,110 +1678,103 @@ static void flush_inline_data(struct f2fs_sb_info *sbi, nid_t ino) iput(inode); } -static struct folio *last_fsync_dnode(struct f2fs_sb_info *sbi, nid_t ino) +static struct f2fs_cached_block *last_fsync_dnode(struct f2fs_sb_info *sbi, nid_t ino) { - pgoff_t index; - struct folio_batch fbatch; - struct folio *last_folio = NULL; - int nr_folios; - - folio_batch_init(&fbatch); - index = 0; + pgoff_t index = 0; + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; + struct f2fs_cached_block *last_entry = NULL; + unsigned int nr; - while ((nr_folios = filemap_get_folios_tag(NODE_MAPPING(sbi), &index, - (pgoff_t)-1, PAGECACHE_TAG_DIRTY, - &fbatch))) { + while ((nr = f2fs_cache_gang_lookup_tag(NODE_CACHE(sbi), + entries, &index, F2FS_ONSTACK_CACHES, + F2FS_CACHE_TAG_DIRTY))) { int i; - for (i = 0; i < nr_folios; i++) { - struct folio *folio = fbatch.folios[i]; + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; if (unlikely(f2fs_cp_error(sbi))) { - f2fs_folio_put(last_folio, false); - folio_batch_release(&fbatch); + f2fs_put_cache(last_entry, false); + f2fs_cache_gang_release(entries, nr); return ERR_PTR(-EIO); } - if (!IS_DNODE(folio) || !is_cold_node(folio)) + if (!IS_DNODE(entry) || !is_cold_node(entry)) continue; - if (ino_of_node(folio) != ino) + if (ino_of_node(entry) != ino) continue; - folio_lock(folio); + f2fs_lock_cache(entry); - if (unlikely(!is_node_folio(folio))) { + if (unlikely(!f2fs_is_node_cache(entry))) { continue_unlock: - folio_unlock(folio); + f2fs_unlock_cache(entry); continue; } - if (ino_of_node(folio) != ino) + if (ino_of_node(entry) != ino) goto continue_unlock; - if (!folio_test_dirty(folio)) { + if (!f2fs_cache_test_dirty(entry)) { /* someone wrote it for us */ goto continue_unlock; } - if (last_folio) - f2fs_folio_put(last_folio, false); + if (last_entry) + f2fs_put_cache(last_entry, false); - folio_get(folio); - last_folio = folio; - folio_unlock(folio); + f2fs_cache_get(entry); + last_entry = entry; + f2fs_unlock_cache(entry); } - folio_batch_release(&fbatch); + f2fs_cache_gang_release(entries, nr); cond_resched(); } - return last_folio; + return last_entry; } -static bool __write_node_folio(struct folio *folio, bool atomic, bool do_fsync, - bool *submitted, struct writeback_control *wbc, - bool do_balance, enum iostat_type io_type, - unsigned int *seq_id) +static bool __write_node_cache(struct f2fs_cached_block *entry, + bool atomic, bool do_fsync, bool *submitted, + bool sync, bool do_balance, + enum iostat_type io_type, unsigned int *seq_id) { - struct f2fs_sb_info *sbi = F2FS_F_SB(folio); + struct f2fs_sb_info *sbi = entry->cache->sbi; nid_t nid; struct node_info ni; struct f2fs_io_info fio = { .sbi = sbi, - .ino = ino_of_node(folio), + .ino = ino_of_node(entry), .type = NODE, .op = REQ_OP_WRITE, - .op_flags = wbc_to_write_flags(wbc), - .folio = folio, + .op_flags = sync ? REQ_SYNC : REQ_BACKGROUND, + .cache_entry = entry, .encrypted_page = NULL, .submitted = 0, .io_type = io_type, - .io_wbc = wbc, + .is_cache = true, }; struct f2fs_lock_context lc; unsigned int seq; - trace_f2fs_writepage(folio, NODE); - if (unlikely(f2fs_cp_error(sbi))) { /* keep node pages in remount-ro mode */ if (F2FS_OPTION(sbi).errors == MOUNT_ERRORS_READONLY) goto redirty_out; - folio_clear_uptodate(folio); - dec_page_count(sbi, F2FS_DIRTY_NODES); - folio_unlock(folio); + f2fs_force_clear_cache_dirty(entry); + f2fs_unlock_cache(entry); return true; } if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING))) goto redirty_out; - if (!is_sbi_flag_set(sbi, SBI_CP_DISABLED) && - wbc->sync_mode == WB_SYNC_NONE && - IS_DNODE(folio) && is_cold_node(folio)) + if (!is_sbi_flag_set(sbi, SBI_CP_DISABLED) && !sync && + IS_DNODE(entry) && is_cold_node(entry)) goto redirty_out; /* get old block addr of this node page */ - nid = nid_of_node(folio); + nid = nid_of_node(entry); - if (f2fs_sanity_check_node_footer(sbi, folio, folio->index, + if (f2fs_sanity_check_node_footer(sbi, entry, entry->index, NODE_TYPE_REGULAR, false)) { fserror_report_metadata(sbi->sb, -EFSCORRUPTED, GFP_NOFS); f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_CORRUPTED_NID); @@ -1802,10 +1788,10 @@ static bool __write_node_folio(struct folio *folio, bool atomic, bool do_fsync, /* This page is already truncated */ if (unlikely(ni.blk_addr == NULL_ADDR)) { - folio_clear_uptodate(folio); + f2fs_cache_clear_uptodate(entry); dec_page_count(sbi, F2FS_DIRTY_NODES); f2fs_up_read_trace(&sbi->node_write, &lc); - folio_unlock(folio); + f2fs_unlock_cache(entry); return true; } @@ -1819,28 +1805,28 @@ static bool __write_node_folio(struct folio *folio, bool atomic, bool do_fsync, if (atomic && !test_opt(sbi, NOBARRIER)) fio.op_flags |= REQ_PREFLUSH | REQ_FUA; - set_dentry_mark(folio, false); - set_fsync_mark(folio, do_fsync); - if (IS_INODE(folio) && (atomic || is_fsync_dnode(folio))) - set_dentry_mark(folio, - f2fs_need_dentry_mark(sbi, ino_of_node(folio))); + set_dentry_mark(entry, false); + set_fsync_mark(entry, do_fsync); + if (IS_INODE(entry) && (atomic || is_fsync_dnode(entry))) + set_dentry_mark(entry, + f2fs_need_dentry_mark(sbi, ino_of_node(entry))); /* should add to global list before clearing PAGECACHE status */ - if (f2fs_in_warm_node_list(folio)) { - seq = f2fs_add_fsync_node_entry(sbi, folio); + if (f2fs_in_warm_node_list(entry)) { + seq = f2fs_add_fsync_node_entry(sbi, entry); if (seq_id) *seq_id = seq; } - folio_start_writeback(folio); + f2fs_start_cache_writeback(entry); fio.old_blkaddr = ni.blk_addr; f2fs_do_write_node_page(nid, &fio); - set_node_addr(sbi, &ni, fio.new_blkaddr, is_fsync_dnode(folio)); + set_node_addr(sbi, &ni, fio.new_blkaddr, is_fsync_dnode(entry)); dec_page_count(sbi, F2FS_DIRTY_NODES); f2fs_up_read_trace(&sbi->node_write, &lc); - folio_unlock(folio); + f2fs_unlock_cache(entry); if (unlikely(f2fs_cp_error(sbi))) { f2fs_submit_merged_write(sbi, NODE); @@ -1854,171 +1840,166 @@ static bool __write_node_folio(struct folio *folio, bool atomic, bool do_fsync, return true; redirty_out: - folio_redirty_for_writepage(wbc, folio); - folio_unlock(folio); + f2fs_cache_set_dirty(entry); + f2fs_unlock_cache(entry); return false; } -int f2fs_write_single_node_folio(struct folio *node_folio, int sync_mode, +int f2fs_write_node_cache(struct f2fs_cached_block *node_entry, int sync_mode, bool mark_dirty, enum iostat_type io_type) { int err = 0; - struct writeback_control wbc = { - .sync_mode = WB_SYNC_ALL, - .nr_to_write = 1, - }; if (!sync_mode) { /* set page dirty and write it */ - if (!folio_test_writeback(node_folio)) - folio_mark_dirty(node_folio); - goto out_folio; + if (!f2fs_cache_test_writeback(node_entry)) + f2fs_mark_cache_dirty(node_entry); + goto out_entry; } - f2fs_folio_wait_writeback(node_folio, NODE, true, true); + f2fs_cache_wait_writeback(node_entry); if (mark_dirty) - folio_mark_dirty(node_folio); - else if (!folio_test_dirty(node_folio)) - goto out_folio; + f2fs_mark_cache_dirty(node_entry); + else if (!f2fs_cache_test_dirty(node_entry)) + goto out_entry; - if (!folio_clear_dirty_for_io(node_folio)) { + if (!f2fs_clear_cache_dirty(node_entry)) { err = -EAGAIN; - goto out_folio; + goto out_entry; } - if (!__write_node_folio(node_folio, false, false, NULL, - &wbc, false, io_type, NULL)) + if (!__write_node_cache(node_entry, false, false, NULL, + true, false, io_type, NULL)) err = -EAGAIN; - goto release_folio; -out_folio: - folio_unlock(node_folio); -release_folio: - f2fs_folio_put(node_folio, false); + goto release_entry; +out_entry: + f2fs_unlock_cache(node_entry); +release_entry: + f2fs_put_cache(node_entry, false); return err; } -int f2fs_move_node_folio(struct folio *node_folio, int gc_type) +int f2fs_move_node_cache(struct f2fs_cached_block *entry, int gc_type) { - return f2fs_write_single_node_folio(node_folio, gc_type == FG_GC, + return f2fs_write_node_cache(entry, gc_type == FG_GC, true, FS_GC_NODE_IO); } -int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode, - struct writeback_control *wbc, bool atomic, - unsigned int *seq_id) +int f2fs_fsync_node_caches(struct f2fs_sb_info *sbi, struct inode *inode, + bool atomic, unsigned int *seq_id) { pgoff_t index; - struct folio_batch fbatch; + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; int ret = 0; - struct folio *last_folio = NULL; + struct f2fs_cached_block *last_entry = NULL; bool marked = false; nid_t ino = inode->i_ino; - int nr_folios; + int nr; int nwritten = 0; if (atomic) { - last_folio = last_fsync_dnode(sbi, ino); - if (IS_ERR_OR_NULL(last_folio)) - return PTR_ERR_OR_ZERO(last_folio); + last_entry = last_fsync_dnode(sbi, ino); + if (IS_ERR_OR_NULL(last_entry)) + return PTR_ERR_OR_ZERO(last_entry); } retry: - folio_batch_init(&fbatch); index = 0; - while ((nr_folios = filemap_get_folios_tag(NODE_MAPPING(sbi), &index, - (pgoff_t)-1, PAGECACHE_TAG_DIRTY, - &fbatch))) { + while ((nr = f2fs_cache_gang_lookup_tag(NODE_CACHE(sbi), + entries, &index, F2FS_ONSTACK_CACHES, + F2FS_CACHE_TAG_DIRTY))) { int i; - for (i = 0; i < nr_folios; i++) { - struct folio *folio = fbatch.folios[i]; + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; bool submitted = false; bool do_fsync = false; if (unlikely(f2fs_cp_error(sbi))) { - f2fs_folio_put(last_folio, false); - folio_batch_release(&fbatch); + f2fs_put_cache(last_entry, false); + f2fs_cache_gang_release(entries, nr); ret = -EIO; goto out; } - if (!IS_DNODE(folio) || !is_cold_node(folio)) + if (!IS_DNODE(entry) || !is_cold_node(entry)) continue; - if (ino_of_node(folio) != ino) + if (ino_of_node(entry) != ino) continue; - folio_lock(folio); + f2fs_lock_cache(entry); - if (unlikely(!is_node_folio(folio))) { + if (unlikely(!f2fs_is_node_cache(entry))) { continue_unlock: - folio_unlock(folio); + f2fs_unlock_cache(entry); continue; } - if (ino_of_node(folio) != ino) + if (ino_of_node(entry) != ino) goto continue_unlock; - if (!folio_test_dirty(folio) && folio != last_folio) { + if (!f2fs_cache_test_dirty(entry) && entry != last_entry) { /* someone wrote it for us */ goto continue_unlock; } - f2fs_folio_wait_writeback(folio, NODE, true, true); + f2fs_cache_wait_writeback(entry); - if (!atomic || folio == last_folio) { + if (!atomic || entry == last_entry) { do_fsync = true; percpu_counter_inc(&sbi->rf_node_block_count); - if (IS_INODE(folio)) { + if (IS_INODE(entry)) { if (is_inode_flag_set(inode, FI_DIRTY_INODE)) - f2fs_update_inode(inode, folio); + f2fs_update_inode(inode, entry); } /* may be written by other thread */ - if (!folio_test_dirty(folio)) - folio_mark_dirty(folio); + if (!f2fs_cache_test_dirty(entry)) + f2fs_mark_cache_dirty(entry); } - if (!folio_clear_dirty_for_io(folio)) + if (!f2fs_clear_cache_dirty(entry)) goto continue_unlock; - if (!__write_node_folio(folio, atomic && - folio == last_folio, + if (!__write_node_cache(entry, atomic && + entry == last_entry, do_fsync, &submitted, - wbc, true, FS_NODE_IO, + true, true, FS_NODE_IO, seq_id)) { - f2fs_folio_put(last_folio, false); - folio_batch_release(&fbatch); + f2fs_put_cache(last_entry, false); + f2fs_cache_gang_release(entries, nr); ret = -EIO; goto out; } if (submitted) nwritten++; - if (folio == last_folio) { - f2fs_folio_put(folio, false); - folio_batch_release(&fbatch); + if (entry == last_entry) { + f2fs_put_cache(entry, false); + f2fs_cache_gang_release(entries, nr); marked = true; goto out; } } - folio_batch_release(&fbatch); + f2fs_cache_gang_release(entries, nr); cond_resched(); } if (atomic && !marked) { f2fs_debug(sbi, "Retry to write fsync mark: ino=%u, idx=%lx", - ino, last_folio->index); - folio_lock(last_folio); - f2fs_folio_wait_writeback(last_folio, NODE, true, true); - folio_mark_dirty(last_folio); - folio_unlock(last_folio); + ino, last_entry->index); + f2fs_lock_cache(last_entry); + f2fs_cache_wait_writeback(last_entry); + f2fs_mark_cache_dirty(last_entry); + f2fs_unlock_cache(last_entry); goto retry; } out: if (nwritten) - f2fs_submit_merged_write_cond(sbi, NULL, NULL, ino, NODE); + f2fs_submit_merged_write_cache(sbi, NULL, ino, NODE); return ret; } + static int f2fs_match_ino(struct inode *inode, u64 ino, void *data) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); @@ -2043,18 +2024,18 @@ static int f2fs_match_ino(struct inode *inode, u64 ino, void *data) return 1; } -static bool flush_dirty_inode(struct folio *folio) +static bool flush_dirty_cache_inode(struct f2fs_cached_block *entry) { - struct f2fs_sb_info *sbi = F2FS_F_SB(folio); + struct f2fs_sb_info *sbi = entry->cache->sbi; struct inode *inode; - nid_t ino = ino_of_node(folio); + nid_t ino = ino_of_node(entry); inode = find_inode_nowait(sbi->sb, ino, f2fs_match_ino, NULL); if (!inode) return false; - f2fs_update_inode(inode, folio); - folio_unlock(folio); + f2fs_update_inode(inode, entry); + f2fs_unlock_cache(entry); iput(inode); return true; @@ -2063,72 +2044,71 @@ static bool flush_dirty_inode(struct folio *folio) void f2fs_flush_inline_data(struct f2fs_sb_info *sbi) { pgoff_t index = 0; - struct folio_batch fbatch; - int nr_folios; + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; + unsigned int nr; - folio_batch_init(&fbatch); - - while ((nr_folios = filemap_get_folios_tag(NODE_MAPPING(sbi), &index, - (pgoff_t)-1, PAGECACHE_TAG_DIRTY, - &fbatch))) { + while ((nr = f2fs_cache_gang_lookup_tag(NODE_CACHE(sbi), + entries, &index, F2FS_ONSTACK_CACHES, + F2FS_CACHE_TAG_DIRTY))) { int i; - for (i = 0; i < nr_folios; i++) { - struct folio *folio = fbatch.folios[i]; + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; - if (!IS_INODE(folio)) + if (!IS_INODE(entry)) continue; - folio_lock(folio); + f2fs_lock_cache(entry); - if (unlikely(!is_node_folio(folio))) + if (unlikely(!f2fs_is_node_cache(entry))) goto unlock; - if (!folio_test_dirty(folio)) + if (!f2fs_cache_test_dirty(entry)) goto unlock; /* flush inline_data, if it's async context. */ - if (folio_test_f2fs_inline(folio)) { - folio_clear_f2fs_inline(folio); - folio_unlock(folio); - flush_inline_data(sbi, ino_of_node(folio)); + if (f2fs_cache_test_inline(entry)) { + f2fs_cache_clear_inline(entry); + f2fs_unlock_cache(entry); + flush_inline_data(sbi, ino_of_node(entry)); continue; } unlock: - folio_unlock(folio); + f2fs_unlock_cache(entry); } - folio_batch_release(&fbatch); + f2fs_cache_gang_release(entries, nr); cond_resched(); } } -int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, - struct writeback_control *wbc, - bool do_balance, enum iostat_type io_type) +int f2fs_writeback_node_caches(struct f2fs_sb_info *sbi, long nr_to_write, + bool sync, bool do_balance, enum iostat_type io_type) { pgoff_t index; - struct folio_batch fbatch; + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; int step = 0; int nwritten = 0; int ret = 0; - int nr_folios, done = 0; - - folio_batch_init(&fbatch); + int nr, done = 0; next_step: index = 0; - while (!done && (nr_folios = filemap_get_folios_tag(NODE_MAPPING(sbi), - &index, (pgoff_t)-1, PAGECACHE_TAG_DIRTY, - &fbatch))) { + while (!done && (nr = f2fs_cache_gang_lookup_tag(NODE_CACHE(sbi), + entries, &index, F2FS_ONSTACK_CACHES, + F2FS_CACHE_TAG_DIRTY))) { int i; - for (i = 0; i < nr_folios; i++) { - struct folio *folio = fbatch.folios[i]; + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; bool submitted = false; + if (!sync && unlikely(freezing(current))) { + done = 1; + break; + } + /* give a priority to WB_SYNC threads */ - if (atomic_read(&sbi->wb_sync_req[NODE]) && - wbc->sync_mode == WB_SYNC_NONE) { + if (atomic_read(&sbi->wb_sync_req[NODE]) && !sync) { done = 1; break; } @@ -2139,27 +2119,27 @@ int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, * 1. dentry dnodes * 2. file dnodes */ - if (step == 0 && IS_DNODE(folio)) + if (step == 0 && IS_DNODE(entry)) continue; - if (step == 1 && (!IS_DNODE(folio) || - is_cold_node(folio))) + if (step == 1 && (!IS_DNODE(entry) || + is_cold_node(entry))) continue; - if (step == 2 && (!IS_DNODE(folio) || - !is_cold_node(folio))) + if (step == 2 && (!IS_DNODE(entry) || + !is_cold_node(entry))) continue; lock_node: - if (wbc->sync_mode == WB_SYNC_ALL) - folio_lock(folio); - else if (!folio_trylock(folio)) + if (sync) + f2fs_lock_cache(entry); + else if (!f2fs_trylock_cache(entry)) continue; - if (unlikely(!is_node_folio(folio))) { + if (unlikely(!f2fs_is_node_cache(entry))) { continue_unlock: - folio_unlock(folio); + f2fs_unlock_cache(entry); continue; } - if (!folio_test_dirty(folio)) { + if (!f2fs_cache_test_dirty(entry)) { /* someone wrote it for us */ goto continue_unlock; } @@ -2169,38 +2149,38 @@ int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, goto write_node; /* flush inline_data */ - if (folio_test_f2fs_inline(folio)) { - folio_clear_f2fs_inline(folio); - folio_unlock(folio); - flush_inline_data(sbi, ino_of_node(folio)); + if (f2fs_cache_test_inline(entry)) { + f2fs_cache_clear_inline(entry); + f2fs_unlock_cache(entry); + flush_inline_data(sbi, ino_of_node(entry)); goto lock_node; } /* flush dirty inode */ - if (IS_INODE(folio) && flush_dirty_inode(folio)) + if (IS_INODE(entry) && flush_dirty_cache_inode(entry)) goto lock_node; write_node: - f2fs_folio_wait_writeback(folio, NODE, true, true); + f2fs_cache_wait_writeback(entry); - if (!folio_clear_dirty_for_io(folio)) + if (!f2fs_clear_cache_dirty(entry)) goto continue_unlock; - if (!__write_node_folio(folio, false, false, &submitted, - wbc, do_balance, io_type, NULL)) { - folio_batch_release(&fbatch); + if (!__write_node_cache(entry, false, false, &submitted, + sync, do_balance, io_type, NULL)) { + f2fs_cache_gang_release(entries, nr); ret = -EIO; goto out; } if (submitted) nwritten++; - if (--wbc->nr_to_write == 0) + if (--nr_to_write == 0) break; } - folio_batch_release(&fbatch); + f2fs_cache_gang_release(entries, nr); cond_resched(); - if (wbc->nr_to_write == 0) { + if (nr_to_write == 0) { step = 2; break; } @@ -2208,7 +2188,7 @@ int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, if (step < 2) { if (!is_sbi_flag_set(sbi, SBI_CP_DISABLED) && - wbc->sync_mode == WB_SYNC_NONE && step == 1) + !sync && step == 1) goto out; step++; goto next_step; @@ -2222,7 +2202,7 @@ int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, return ret; } -int f2fs_wait_on_node_pages_writeback(struct f2fs_sb_info *sbi, +int f2fs_wait_on_node_caches_writeback(struct f2fs_sb_info *sbi, unsigned int seq_id) { struct fsync_node_entry *fn; @@ -2231,7 +2211,7 @@ int f2fs_wait_on_node_pages_writeback(struct f2fs_sb_info *sbi, unsigned int cur_seq_id = 0; while (seq_id && cur_seq_id < seq_id) { - struct folio *folio; + struct f2fs_cached_block *entry; spin_lock_irqsave(&sbi->fsync_node_lock, flags); if (list_empty(head)) { @@ -2244,94 +2224,48 @@ int f2fs_wait_on_node_pages_writeback(struct f2fs_sb_info *sbi, break; } cur_seq_id = fn->seq_id; - folio = fn->folio; - folio_get(folio); + entry = fn->entry; + f2fs_cache_get(entry); spin_unlock_irqrestore(&sbi->fsync_node_lock, flags); - f2fs_folio_wait_writeback(folio, NODE, true, false); + f2fs_cache_wait_writeback(entry); - folio_put(folio); + f2fs_put_cache(entry, false); } - return filemap_check_errors(NODE_MAPPING(sbi)); + return f2fs_cp_error(sbi) ? -EIO : 0; } -static int f2fs_write_node_pages(struct address_space *mapping, - struct writeback_control *wbc) +int f2fs_write_node_caches(struct f2fs_sb_info *sbi) { - struct f2fs_sb_info *sbi = F2FS_M_SB(mapping); struct blk_plug plug; - long diff; + long nr_to_write = LONG_MAX; if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING))) - goto skip_write; + return -EAGAIN; /* balancing f2fs's metadata in background */ f2fs_balance_fs_bg(sbi, true); /* collect a number of dirty node pages and write together */ - if (wbc->sync_mode != WB_SYNC_ALL && - get_pages(sbi, F2FS_DIRTY_NODES) < + if (get_pages(sbi, F2FS_DIRTY_NODES) < nr_pages_to_skip(sbi, NODE)) - goto skip_write; + return -EAGAIN; - if (wbc->sync_mode == WB_SYNC_ALL) - atomic_inc(&sbi->wb_sync_req[NODE]); - else if (atomic_read(&sbi->wb_sync_req[NODE])) { + if (atomic_read(&sbi->wb_sync_req[NODE])) { /* to avoid potential deadlock */ if (current->plug) blk_finish_plug(current->plug); - goto skip_write; + return -EAGAIN; } - trace_f2fs_writepages(mapping->host, wbc, NODE); - - diff = nr_pages_to_write(sbi, NODE, wbc); + nr_to_write = adjust_flush_cache_number(sbi, NODE); blk_start_plug(&plug); - f2fs_sync_node_pages(sbi, wbc, true, FS_NODE_IO); + f2fs_writeback_node_caches(sbi, nr_to_write, false, true, FS_NODE_IO); blk_finish_plug(&plug); - wbc->nr_to_write = max((long)0, wbc->nr_to_write - diff); - - if (wbc->sync_mode == WB_SYNC_ALL) - atomic_dec(&sbi->wb_sync_req[NODE]); - return 0; - -skip_write: - wbc->pages_skipped += get_pages(sbi, F2FS_DIRTY_NODES); - trace_f2fs_writepages(mapping->host, wbc, NODE); return 0; } -static bool f2fs_dirty_node_folio(struct address_space *mapping, - struct folio *folio) -{ - trace_f2fs_set_page_dirty(folio, NODE); - - if (!folio_test_uptodate(folio)) - folio_mark_uptodate(folio); -#ifdef CONFIG_F2FS_CHECK_FS - if (IS_INODE(folio)) - f2fs_inode_chksum_set(F2FS_M_SB(mapping), folio); -#endif - if (filemap_dirty_folio(mapping, folio)) { - inc_page_count(F2FS_M_SB(mapping), F2FS_DIRTY_NODES); - folio_set_f2fs_reference(folio); - return true; - } - return false; -} - -/* - * Structure of the f2fs node operations - */ -const struct address_space_operations f2fs_node_aops = { - .writepages = f2fs_write_node_pages, - .dirty_folio = f2fs_dirty_node_folio, - .invalidate_folio = f2fs_invalidate_folio, - .release_folio = f2fs_release_folio, - .migrate_folio = filemap_migrate_folio, -}; - static struct free_nid *__lookup_free_nid_list(struct f2fs_nm_info *nm_i, nid_t n) { @@ -2829,12 +2763,12 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct f2fs_cached_block *ent { void *src_addr, *dst_addr; size_t inline_size; - struct folio *ifolio; + struct f2fs_cached_block *ientry; struct f2fs_inode *ri; - ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); ri = &CACHED_NODE(entry)->i; if (ri->i_inline & F2FS_INLINE_XATTR) { @@ -2850,15 +2784,15 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct f2fs_cached_block *ent goto update_inode; } - dst_addr = inline_xattr_addr(inode, ifolio); - src_addr = inline_xattr_addr(inode, cache_folio(entry)); + dst_addr = inline_xattr_addr(inode, ientry); + src_addr = inline_xattr_addr(inode, entry); inline_size = inline_xattr_size(inode); - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_cache_wait_writeback(ientry); memcpy(dst_addr, src_addr, inline_size); update_inode: - f2fs_update_inode(inode, ifolio); - f2fs_folio_put(ifolio, true); + f2fs_update_inode(inode, ientry); + f2fs_put_cache(ientry, true); return 0; } @@ -2869,7 +2803,7 @@ int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry nid_t new_xnid; struct dnode_of_data dn; struct node_info ni; - struct folio *xfolio; + struct f2fs_cached_block *xentry; int err; if (!prev_xnid) @@ -2890,10 +2824,10 @@ int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry return -ENOSPC; set_new_dnode(&dn, inode, NULL, NULL, new_xnid); - xfolio = f2fs_new_node_folio(&dn, XATTR_NODE_OFFSET); - if (IS_ERR(xfolio)) { + xentry = f2fs_new_node_cache(&dn, XATTR_NODE_OFFSET); + if (IS_ERR(xentry)) { f2fs_alloc_nid_failed(sbi, new_xnid); - return PTR_ERR(xfolio); + return PTR_ERR(xentry); } f2fs_alloc_nid_done(sbi, new_xnid); @@ -2901,11 +2835,11 @@ int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry /* 3: update and set xattr node page dirty */ if (entry) { - memcpy(F2FS_NODE(xfolio), CACHED_NODE(entry), + memcpy(CACHED_NODE(xentry), CACHED_NODE(entry), VALID_XATTR_BLOCK_SIZE); - folio_mark_dirty(xfolio); + f2fs_mark_cache_dirty(xentry); } - f2fs_folio_put(xfolio, true); + f2fs_put_cache(xentry, true); return 0; } @@ -2913,9 +2847,9 @@ int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) { struct f2fs_inode *src, *dst; - nid_t ino = ino_of_node(cache_folio(entry)); + nid_t ino = ino_of_node(entry); struct node_info old_ni, new_ni; - struct folio *ifolio; + struct f2fs_cached_block *ientry; int err; err = f2fs_get_node_info(sbi, ino, &old_ni, false); @@ -2925,8 +2859,8 @@ int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block * if (unlikely(old_ni.blk_addr != NULL_ADDR)) return -EINVAL; retry: - ifolio = f2fs_grab_cache_folio(NODE_MAPPING(sbi), ino, false); - if (IS_ERR(ifolio)) { + ientry = f2fs_grab_node_cache(sbi, ino); + if (IS_ERR(ientry)) { memalloc_retry_wait(GFP_NOFS); goto retry; } @@ -2934,13 +2868,12 @@ int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block * /* Should not use this inode from free nid list */ remove_free_nid(sbi, ino); - if (!folio_test_uptodate(ifolio)) - folio_mark_uptodate(ifolio); - fill_node_footer(ifolio, ino, ino, 0, true); - set_cold_node(ifolio, false); + f2fs_cache_set_uptodate(ientry); + fill_node_footer(ientry, ino, ino, 0, true); + set_cold_node(ientry, false); src = &CACHED_NODE(entry)->i; - dst = F2FS_INODE(ifolio); + dst = F2FS_INODE(ientry); memcpy(dst, src, offsetof(struct f2fs_inode, i_ext)); dst->i_size = 0; @@ -2976,8 +2909,8 @@ int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block * WARN_ON(1); set_node_addr(sbi, &new_ni, NEW_ADDR, false); inc_valid_inode_count(sbi); - folio_mark_dirty(ifolio); - f2fs_folio_put(ifolio, true); + f2fs_mark_cache_dirty(ientry); + f2fs_put_cache(ientry, true); return 0; } diff --git a/fs/f2fs/node.h b/fs/f2fs/node.h index de8dcd5d4392..e37a0cb708f9 100644 --- a/fs/f2fs/node.h +++ b/fs/f2fs/node.h @@ -240,41 +240,36 @@ static inline void set_to_next_nat(struct f2fs_nm_info *nm_i, nid_t start_nid) #endif } -static inline nid_t ino_of_node(const struct folio *node_folio) +static inline nid_t ino_of_node(const struct f2fs_cached_block *entry) { - struct f2fs_node *rn = F2FS_NODE(node_folio); - return le32_to_cpu(rn->footer.ino); + return le32_to_cpu(CACHED_NODE(entry)->footer.ino); } -static inline nid_t nid_of_node(const struct folio *node_folio) +static inline nid_t nid_of_node(const struct f2fs_cached_block *entry) { - struct f2fs_node *rn = F2FS_NODE(node_folio); - return le32_to_cpu(rn->footer.nid); + return le32_to_cpu(CACHED_NODE(entry)->footer.nid); } -static inline unsigned int ofs_of_node(const struct folio *node_folio) +static inline unsigned int ofs_of_node(const struct f2fs_cached_block *entry) { - struct f2fs_node *rn = F2FS_NODE(node_folio); - unsigned flag = le32_to_cpu(rn->footer.flag); + unsigned int flag = le32_to_cpu(CACHED_NODE(entry)->footer.flag); return flag >> OFFSET_BIT_SHIFT; } -static inline __u64 cpver_of_node(const struct folio *node_folio) +static inline __u64 cpver_of_node(const struct f2fs_cached_block *entry) { - struct f2fs_node *rn = F2FS_NODE(node_folio); - return le64_to_cpu(rn->footer.cp_ver); + return le64_to_cpu(CACHED_NODE(entry)->footer.cp_ver); } -static inline block_t next_blkaddr_of_node(const struct folio *node_folio) +static inline block_t next_blkaddr_of_node(const struct f2fs_cached_block *entry) { - struct f2fs_node *rn = F2FS_NODE(node_folio); - return le32_to_cpu(rn->footer.next_blkaddr); + return le32_to_cpu(CACHED_NODE(entry)->footer.next_blkaddr); } -static inline void fill_node_footer(const struct folio *folio, nid_t nid, +static inline void fill_node_footer(struct f2fs_cached_block *entry, nid_t nid, nid_t ino, unsigned int ofs, bool reset) { - struct f2fs_node *rn = F2FS_NODE(folio); + struct f2fs_node *rn = CACHED_NODE(entry); unsigned int old_flag = 0; if (reset) @@ -290,18 +285,16 @@ static inline void fill_node_footer(const struct folio *folio, nid_t nid, (old_flag & OFFSET_BIT_MASK)); } -static inline void copy_node_footer(const struct folio *dst, - const struct folio *src) +static inline void copy_node_footer(struct f2fs_cached_block *dst, + const struct f2fs_cached_block *src) { - struct f2fs_node *src_rn = F2FS_NODE(src); - struct f2fs_node *dst_rn = F2FS_NODE(dst); - memcpy(&dst_rn->footer, &src_rn->footer, sizeof(struct node_footer)); + memcpy(&CACHED_NODE(dst)->footer, &CACHED_NODE(src)->footer, sizeof(struct node_footer)); } -static inline void fill_node_footer_blkaddr(struct folio *folio, block_t blkaddr) +static inline void fill_node_footer_blkaddr(struct f2fs_cached_block *entry, block_t blkaddr) { - struct f2fs_checkpoint *ckpt = F2FS_CKPT(F2FS_F_SB(folio)); - struct f2fs_node *rn = F2FS_NODE(folio); + struct f2fs_node *rn = CACHED_NODE(entry); + struct f2fs_checkpoint *ckpt = F2FS_CKPT(entry->cache->sbi); __u64 cp_ver = cur_cp_version(ckpt); if (__is_set_ckpt_flags(ckpt, CP_CRC_RECOVERY_FLAG)) @@ -311,19 +304,20 @@ static inline void fill_node_footer_blkaddr(struct folio *folio, block_t blkaddr rn->footer.next_blkaddr = cpu_to_le32(blkaddr); } -static inline bool is_recoverable_dnode(struct f2fs_sb_info *sbi, const struct folio *folio) +static inline bool is_recoverable_dnode(struct f2fs_sb_info *sbi, + const struct f2fs_cached_block *entry) { struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); __u64 cp_ver = cur_cp_version(ckpt); /* Don't care crc part, if fsck.f2fs sets it. */ if (__is_set_ckpt_flags(ckpt, CP_NOCRC_RECOVERY_FLAG)) - return (cp_ver << 32) == (cpver_of_node(folio) << 32); + return (cp_ver << 32) == (cpver_of_node(entry) << 32); if (__is_set_ckpt_flags(ckpt, CP_CRC_RECOVERY_FLAG)) cp_ver |= (cur_cp_crc(ckpt) << 32); - return cp_ver == cpver_of_node(folio); + return cp_ver == cpver_of_node(entry); } /* @@ -347,9 +341,9 @@ static inline bool is_recoverable_dnode(struct f2fs_sb_info *sbi, const struct f * `- indirect node ((6 + 2N) + (N - 1)(N + 1)) * `- direct node */ -static inline bool IS_DNODE(const struct folio *node_folio) +static inline bool IS_DNODE(const struct f2fs_cached_block *entry) { - unsigned int ofs = ofs_of_node(node_folio); + unsigned int ofs = ofs_of_node(entry); if (f2fs_has_xattr_block(ofs)) return true; @@ -365,23 +359,23 @@ static inline bool IS_DNODE(const struct folio *node_folio) return true; } -static inline int set_nid(struct folio *folio, int off, nid_t nid, bool i) +static inline bool set_nid(struct f2fs_cached_block *entry, int off, nid_t nid, bool i) { - struct f2fs_node *rn = F2FS_NODE(folio); + struct f2fs_node *rn = CACHED_NODE(entry); + __le32 *addr = i ? &rn->i.i_nid[off - NODE_DIR1_BLOCK] : &rn->in.nid[off]; - f2fs_folio_wait_writeback(folio, NODE, true, true); + f2fs_cache_wait_writeback(entry); + if (*addr == cpu_to_le32(nid)) + return false; - if (i) - rn->i.i_nid[off - NODE_DIR1_BLOCK] = cpu_to_le32(nid); - else - rn->in.nid[off] = cpu_to_le32(nid); - return folio_mark_dirty(folio); + *addr = cpu_to_le32(nid); + f2fs_mark_cache_dirty(entry); + return true; } -static inline nid_t get_nid(const struct folio *folio, int off, bool i) +static inline nid_t get_nid(const struct f2fs_cached_block *entry, int off, bool i) { - struct f2fs_node *rn = F2FS_NODE(folio); - + struct f2fs_node *rn = CACHED_NODE(entry); if (i) return le32_to_cpu(rn->i.i_nid[off - NODE_DIR1_BLOCK]); return le32_to_cpu(rn->in.nid[off]); @@ -394,19 +388,18 @@ static inline nid_t get_nid(const struct folio *folio, int off, bool i) * - Mark cold data pages in page cache */ -static inline int is_node(const struct folio *folio, int type) +static inline int is_node(const struct f2fs_cached_block *entry, int type) { - struct f2fs_node *rn = F2FS_NODE(folio); - return le32_to_cpu(rn->footer.flag) & BIT(type); + return le32_to_cpu(CACHED_NODE(entry)->footer.flag) & BIT(type); } -#define is_cold_node(folio) is_node(folio, COLD_BIT_SHIFT) -#define is_fsync_dnode(folio) is_node(folio, FSYNC_BIT_SHIFT) -#define is_dent_dnode(folio) is_node(folio, DENT_BIT_SHIFT) +#define is_cold_node(entry) is_node(entry, COLD_BIT_SHIFT) +#define is_fsync_dnode(entry) is_node(entry, FSYNC_BIT_SHIFT) +#define is_dent_dnode(entry) is_node(entry, DENT_BIT_SHIFT) -static inline void __set_mark(const struct folio *folio, bool mark, int type) +static inline void __set_mark(struct f2fs_cached_block *entry, bool mark, int type) { - struct f2fs_node *rn = F2FS_NODE(folio); + struct f2fs_node *rn = CACHED_NODE(entry); unsigned int flag = le32_to_cpu(rn->footer.flag); if (mark) @@ -416,18 +409,18 @@ static inline void __set_mark(const struct folio *folio, bool mark, int type) rn->footer.flag = cpu_to_le32(flag); } -static inline void set_cold_node(const struct folio *folio, bool is_dir) +static inline void set_cold_node(struct f2fs_cached_block *entry, bool is_dir) { - __set_mark(folio, !is_dir, COLD_BIT_SHIFT); + __set_mark(entry, !is_dir, COLD_BIT_SHIFT); } -static inline void set_mark(struct folio *folio, bool mark, int type) +static inline void set_mark(struct f2fs_cached_block *entry, bool mark, int type) { - __set_mark(folio, mark, type); - + __set_mark(entry, mark, type); #ifdef CONFIG_F2FS_CHECK_FS - f2fs_inode_chksum_set(F2FS_F_SB(folio), folio); + f2fs_inode_chksum_set(entry->cache->sbi, entry); #endif } -#define set_dentry_mark(folio, mark) set_mark(folio, mark, DENT_BIT_SHIFT) -#define set_fsync_mark(folio, mark) set_mark(folio, mark, FSYNC_BIT_SHIFT) + +#define set_dentry_mark(entry, mark) set_mark(entry, mark, DENT_BIT_SHIFT) +#define set_fsync_mark(entry, mark) set_mark(entry, mark, FSYNC_BIT_SHIFT) diff --git a/fs/f2fs/recovery.c b/fs/f2fs/recovery.c index 5cb56b4c1879..1e414b5a205f 100644 --- a/fs/f2fs/recovery.c +++ b/fs/f2fs/recovery.c @@ -190,7 +190,7 @@ static int recover_dentry(struct inode *inode, struct f2fs_cached_block *entry, struct f2fs_dir_entry *de; struct f2fs_filename fname; struct qstr usr_fname; - struct folio *folio; + void *dentry_blk = NULL; struct inode *dir, *einode; struct fsync_inode_entry *fsync_entry; int err = 0; @@ -213,7 +213,8 @@ static int recover_dentry(struct inode *inode, struct f2fs_cached_block *entry, if (err) goto out; retry: - de = __f2fs_find_entry(dir, &fname, &folio); + dentry_blk = NULL; + de = __f2fs_find_entry(dir, &fname, &dentry_blk); if (de && inode->i_ino == le32_to_cpu(de->ino)) goto out_put; @@ -238,11 +239,11 @@ static int recover_dentry(struct inode *inode, struct f2fs_cached_block *entry, iput(einode); goto out_put; } - f2fs_delete_entry(de, folio, dir, einode); + f2fs_delete_entry(de, dentry_blk, dir, einode); iput(einode); goto retry; - } else if (IS_ERR(folio)) { - err = PTR_ERR(folio); + } else if (IS_ERR(dentry_blk)) { + err = PTR_ERR(dentry_blk); } else { err = f2fs_add_dentry(dir, &fname, inode, inode->i_ino, inode->i_mode); @@ -252,11 +253,11 @@ static int recover_dentry(struct inode *inode, struct f2fs_cached_block *entry, goto out; out_put: - f2fs_folio_put(folio, false); + f2fs_put_dentry_block(dentry_blk, false); out: name = recover_printable_name(inode, raw_inode, &name_len); f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, dir = %llu, err = %d", - __func__, ino_of_node(cache_folio(entry)), name_len, name, + __func__, ino_of_node(entry), name_len, name, IS_ERR(dir) ? 0 : dir->i_ino, err); return err; } @@ -357,7 +358,7 @@ static int recover_inode(struct inode *inode, struct f2fs_cached_block *entry) name = recover_printable_name(inode, raw, &name_len); f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, inline = %x", - __func__, ino_of_node(cache_folio(entry)), name_len, name, + __func__, ino_of_node(entry), name_len, name, raw->i_inline); return 0; } @@ -397,16 +398,16 @@ static int sanity_check_node_chain(struct f2fs_sb_info *sbi, block_t blkaddr, if (IS_ERR(entry)) return PTR_ERR(entry); - if (!is_recoverable_dnode(sbi, cache_folio(entry))) { + if (!is_recoverable_dnode(sbi, entry)) { f2fs_put_cache(entry, true); *is_detecting = false; return 0; } ra_blocks = adjust_por_ra_blocks(sbi, ra_blocks, *blkaddr_fast, - next_blkaddr_of_node(cache_folio(entry))); + next_blkaddr_of_node(entry)); - *blkaddr_fast = next_blkaddr_of_node(cache_folio(entry)); + *blkaddr_fast = next_blkaddr_of_node(entry); f2fs_put_cache(entry, true); f2fs_ra_meta_caches_cond(sbi, *blkaddr_fast, ra_blocks); @@ -448,21 +449,21 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head, } rn = CACHED_NODE(entry); - if (!is_recoverable_dnode(sbi, cache_folio(entry))) { + if (!is_recoverable_dnode(sbi, entry)) { f2fs_put_cache(entry, true); break; } - if (!is_fsync_dnode(cache_folio(entry))) + if (!is_fsync_dnode(entry)) goto next; - fsync_entry = get_fsync_inode(head, ino_of_node(cache_folio(entry))); + fsync_entry = get_fsync_inode(head, ino_of_node(entry)); if (!fsync_entry) { bool quota_inode = false; if (!check_only && - IS_INODE(cache_folio(entry)) && - is_dent_dnode(cache_folio(entry))) { + IS_INODE(entry) && + is_dent_dnode(entry)) { err = f2fs_recover_inode_page(sbi, entry); if (err) { f2fs_put_cache(entry, true); @@ -471,7 +472,7 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head, quota_inode = true; } - fsync_entry = add_fsync_inode(sbi, head, ino_of_node(cache_folio(entry)), + fsync_entry = add_fsync_inode(sbi, head, ino_of_node(entry), quota_inode); if (IS_ERR(fsync_entry)) { err = PTR_ERR(fsync_entry); @@ -490,11 +491,11 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head, } fsync_entry->blkaddr = blkaddr; - if (IS_INODE(cache_folio(entry)) && is_dent_dnode(cache_folio(entry))) + if (IS_INODE(entry) && is_dent_dnode(entry)) fsync_entry->last_dentry = blkaddr; next: /* check next segment */ - blkaddr = next_blkaddr_of_node(cache_folio(entry)); + blkaddr = next_blkaddr_of_node(entry); f2fs_put_cache(entry, true); err = sanity_check_node_chain(sbi, blkaddr, &blkaddr_fast, @@ -522,7 +523,7 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, struct f2fs_summary_block *sum_node; struct f2fs_summary sum; struct f2fs_cached_block *entry = NULL; - struct folio *node_folio; + struct f2fs_cached_block *node_entry; struct dnode_of_data tdn = *dn; nid_t ino, nid; struct inode *inode; @@ -555,7 +556,7 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, nid = le32_to_cpu(sum.nid); ofs_in_node = le16_to_cpu(sum.ofs_in_node); - max_addrs = ADDRS_PER_PAGE(dn->node_folio, dn->inode); + max_addrs = ADDRS_PER_PAGE(dn->node_entry, dn->inode); if (ofs_in_node >= max_addrs) { f2fs_err(sbi, "Inconsistent ofs_in_node:%u in summary, ino:%llu, nid:%u, max:%u", ofs_in_node, dn->inode->i_ino, nid, max_addrs); @@ -565,9 +566,9 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, if (dn->inode->i_ino == nid) { tdn.nid = nid; - if (!dn->inode_folio_locked) - folio_lock(dn->inode_folio); - tdn.node_folio = dn->inode_folio; + if (!dn->inode_entry_locked) + f2fs_lock_cache(dn->inode_entry); + tdn.node_entry = dn->inode_entry; tdn.ofs_in_node = ofs_in_node; goto truncate_out; } else if (dn->nid == nid) { @@ -576,13 +577,13 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, } /* Get the node page */ - node_folio = f2fs_get_node_folio(sbi, nid, NODE_TYPE_REGULAR); - if (IS_ERR(node_folio)) - return PTR_ERR(node_folio); + node_entry = f2fs_get_node_cache(sbi, nid, NODE_TYPE_REGULAR); + if (IS_ERR(node_entry)) + return PTR_ERR(node_entry); - offset = ofs_of_node(node_folio); - ino = ino_of_node(node_folio); - f2fs_folio_put(node_folio, true); + offset = ofs_of_node(node_entry); + ino = ino_of_node(node_entry); + f2fs_put_cache(node_entry, true); if (ino != dn->inode->i_ino) { int ret; @@ -608,8 +609,8 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, * if inode page is locked, unlock temporarily, but its reference * count keeps alive. */ - if (ino == dn->inode->i_ino && dn->inode_folio_locked) - folio_unlock(dn->inode_folio); + if (ino == dn->inode->i_ino && dn->inode_entry_locked) + f2fs_unlock_cache(dn->inode_entry); set_new_dnode(&tdn, inode, NULL, NULL, 0); if (f2fs_get_dnode_of_data(&tdn, bidx, LOOKUP_NODE)) @@ -622,15 +623,15 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, out: if (ino != dn->inode->i_ino) iput(inode); - else if (dn->inode_folio_locked) - folio_lock(dn->inode_folio); + else if (dn->inode_entry_locked) + f2fs_lock_cache(dn->inode_entry); return 0; truncate_out: if (f2fs_data_blkaddr(&tdn) == blkaddr) f2fs_truncate_data_blocks_range(&tdn, 1); - if (dn->inode->i_ino == nid && !dn->inode_folio_locked) - folio_unlock(dn->inode_folio); + if (dn->inode->i_ino == nid && !dn->inode_entry_locked) + f2fs_unlock_cache(dn->inode_entry); return 0; } @@ -656,11 +657,11 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, int err = 0, recovered = 0; /* step 1: recover xattr */ - if (IS_INODE(cache_folio(entry))) { + if (IS_INODE(entry)) { err = f2fs_recover_inline_xattr(inode, entry); if (err) goto out; - } else if (f2fs_has_xattr_block(ofs_of_node(cache_folio(entry)))) { + } else if (f2fs_has_xattr_block(ofs_of_node(entry))) { err = f2fs_recover_xattr_data(inode, entry); if (!err) recovered++; @@ -676,8 +677,8 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, } /* step 3: recover data indices */ - start = f2fs_start_bidx_of_node(ofs_of_node(cache_folio(entry)), inode); - end = start + addrs_per_page(inode, IS_INODE(cache_folio(entry))); + start = f2fs_start_bidx_of_node(ofs_of_node(entry), inode); + end = start + addrs_per_page(inode, IS_INODE(entry)); set_new_dnode(&dn, inode, NULL, NULL, 0); retry_dn: @@ -690,18 +691,18 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, goto out; } - f2fs_folio_wait_writeback(dn.node_folio, NODE, true, true); + f2fs_cache_wait_writeback(dn.node_entry); err = f2fs_get_node_info(sbi, dn.nid, &ni, false); if (err) goto err; - f2fs_bug_on(sbi, ni.ino != ino_of_node(cache_folio(entry))); + f2fs_bug_on(sbi, ni.ino != ino_of_node(entry)); - if (ofs_of_node(dn.node_folio) != ofs_of_node(cache_folio(entry))) { + if (ofs_of_node(dn.node_entry) != ofs_of_node(entry)) { f2fs_warn(sbi, "Inconsistent ofs_of_node, ino:%llu, ofs:%u, %u", - inode->i_ino, ofs_of_node(dn.node_folio), - ofs_of_node(cache_folio(entry))); + inode->i_ino, ofs_of_node(dn.node_entry), + ofs_of_node(entry)); err = -EFSCORRUPTED; f2fs_handle_error(sbi, ERROR_INCONSISTENT_FOOTER); fserror_report_file_metadata(dn.inode, err, GFP_NOFS); @@ -712,7 +713,7 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, block_t src, dest; src = f2fs_data_blkaddr(&dn); - dest = data_blkaddr(dn.inode, cache_folio(entry), dn.ofs_in_node); + dest = data_blkaddr(dn.inode, entry, dn.ofs_in_node); if (__is_valid_data_blkaddr(src) && !f2fs_is_valid_blkaddr(sbi, src, META_POR)) { @@ -787,16 +788,16 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, } } - copy_node_footer(dn.node_folio, cache_folio(entry)); - fill_node_footer(dn.node_folio, dn.nid, ni.ino, - ofs_of_node(cache_folio(entry)), false); - folio_mark_dirty(dn.node_folio); + copy_node_footer(dn.node_entry, entry); + fill_node_footer(dn.node_entry, dn.nid, ni.ino, + ofs_of_node(entry), false); + f2fs_mark_cache_dirty(dn.node_entry); err: f2fs_put_dnode(&dn); out: f2fs_notice(sbi, "recover_data: ino = %llx, nid = %x (i_size: %s), " "range (%u, %u), recovered = %d, err = %d", - inode->i_ino, nid_of_node(cache_folio(entry)), + inode->i_ino, nid_of_node(entry), file_keep_isize(inode) ? "keep" : "recover", start, end, recovered, err); return err; @@ -835,13 +836,13 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list, break; } - if (!is_recoverable_dnode(sbi, cache_folio(entry))) { + if (!is_recoverable_dnode(sbi, entry)) { f2fs_put_cache(entry, true); break; } recoverable_dnode++; - fsync_entry = get_fsync_inode(inode_list, ino_of_node(cache_folio(entry))); + fsync_entry = get_fsync_inode(inode_list, ino_of_node(entry)); if (!fsync_entry) goto next; fsynced_dnode++; @@ -850,7 +851,7 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list, * In this case, we can lose the latest inode(x). * So, call recover_inode for the inode update. */ - if (IS_INODE(cache_folio(entry))) { + if (IS_INODE(entry)) { err = recover_inode(fsync_entry->inode, entry); if (err) { f2fs_put_cache(entry, true); @@ -877,10 +878,10 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list, list_move_tail(&fsync_entry->list, tmp_inode_list); next: ra_blocks = adjust_por_ra_blocks(sbi, ra_blocks, blkaddr, - next_blkaddr_of_node(cache_folio(entry))); + next_blkaddr_of_node(entry)); /* check next segment */ - blkaddr = next_blkaddr_of_node(cache_folio(entry)); + blkaddr = next_blkaddr_of_node(entry); f2fs_put_cache(entry, true); f2fs_ra_meta_caches_cond(sbi, blkaddr, ra_blocks); @@ -943,7 +944,7 @@ int f2fs_recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only) f2fs_truncate_meta_caches(sbi, MAIN_BLKADDR(sbi), MAX_BLKADDR(sbi) - MAIN_BLKADDR(sbi)); if (err) { - truncate_inode_pages_final(NODE_MAPPING(sbi)); + f2fs_truncate_node_caches(sbi, 0, ULONG_MAX); f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); } diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c index cbb3a8c9f4ab..3a585fc7c197 100644 --- a/fs/f2fs/segment.c +++ b/fs/f2fs/segment.c @@ -335,7 +335,7 @@ static int __f2fs_commit_atomic_write(struct inode *inode) goto next; } - blen = min((pgoff_t)ADDRS_PER_PAGE(dn.node_folio, cow_inode), + blen = min((pgoff_t)addrs_per_page(cow_inode, IS_INODE(dn.node_entry)), len); index = off; for (i = 0; i < blen; i++, dn.ofs_in_node++, index++) { @@ -3768,7 +3768,8 @@ static int __get_segment_type_4(struct f2fs_io_info *fio) else return CURSEG_COLD_DATA; } else { - if (IS_DNODE(fio->folio) && is_cold_node(fio->folio)) + f2fs_bug_on(fio->sbi, !fio->is_cache); + if (IS_DNODE(fio->cache_entry) && is_cold_node(fio->cache_entry)) return CURSEG_WARM_NODE; else return CURSEG_COLD_NODE; @@ -3826,9 +3827,9 @@ static int __get_segment_type_6(struct f2fs_io_info *fio) return f2fs_rw_hint_to_seg_type(F2FS_I_SB(inode), inode->i_write_hint); } else { - if (IS_DNODE(fio->folio)) - return is_cold_node(fio->folio) ? CURSEG_WARM_NODE : - CURSEG_HOT_NODE; + f2fs_bug_on(fio->sbi, !fio->is_cache); + if (IS_DNODE(fio->cache_entry)) + return is_cold_node(fio->cache_entry) ? CURSEG_WARM_NODE : CURSEG_HOT_NODE; return CURSEG_COLD_NODE; } } @@ -3895,7 +3896,7 @@ static void f2fs_randomize_chunk(struct f2fs_sb_info *sbi, get_random_u32_inclusive(1, sbi->max_fragment_hole); } -int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, struct folio *folio, +int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, block_t old_blkaddr, block_t *new_blkaddr, struct f2fs_summary *sum, int type, struct f2fs_io_info *fio) @@ -4003,10 +4004,10 @@ int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, struct folio *folio, up_write(&sit_i->sentry_lock); - if (folio && IS_NODESEG(curseg->seg_type)) { - fill_node_footer_blkaddr(folio, NEXT_FREE_BLKADDR(sbi, curseg)); - - f2fs_inode_chksum_set(sbi, folio); + if (fio && fio->is_cache && IS_NODESEG(curseg->seg_type)) { + fill_node_footer_blkaddr(fio->cache_entry, + NEXT_FREE_BLKADDR(sbi, curseg)); + f2fs_inode_chksum_set(sbi, fio->cache_entry); } if (fio) { @@ -4094,18 +4095,26 @@ static void do_write_page(struct f2fs_summary *sum, struct f2fs_io_info *fio) if (keep_order) f2fs_down_read(&fio->sbi->io_order_lock); - err = f2fs_allocate_data_block(fio->sbi, folio, fio->old_blkaddr, + err = f2fs_allocate_data_block(fio->sbi, fio->old_blkaddr, &fio->new_blkaddr, sum, type, fio); if (unlikely(err)) { - f2fs_err_ratelimited(fio->sbi, - "%s Failed to allocate data block, ino:%u, index:%lu, type:%d, old_blkaddr:0x%x, new_blkaddr:0x%x, err:%d", - __func__, fio->ino, folio->index, type, - fio->old_blkaddr, fio->new_blkaddr, err); - if (fscrypt_inode_uses_fs_layer_crypto(folio->mapping->host)) - fscrypt_finalize_bounce_page(&fio->encrypted_page); - folio_end_writeback(folio); - if (f2fs_in_warm_node_list(folio)) - f2fs_del_fsync_node_entry(fio->sbi, folio); + if (fio->is_cache) { + f2fs_err_ratelimited(fio->sbi, + "%s Failed to allocate data block, ino:%u, index:%lu, type:%d, old_blkaddr:0x%x, new_blkaddr:0x%x, err:%d", + __func__, fio->ino, fio->cache_entry->index, type, + fio->old_blkaddr, fio->new_blkaddr, err); + f2fs_end_cache_writeback(fio->cache_entry); + if (f2fs_in_warm_node_list(fio->cache_entry)) + f2fs_del_fsync_node_entry(fio->sbi, fio->cache_entry); + } else { + f2fs_err_ratelimited(fio->sbi, + "%s Failed to allocate data block, ino:%u, index:%lu, type:%d, old_blkaddr:0x%x, new_blkaddr:0x%x, err:%d", + __func__, fio->ino, folio->index, type, + fio->old_blkaddr, fio->new_blkaddr, err); + if (fscrypt_inode_uses_fs_layer_crypto(folio->mapping->host)) + fscrypt_finalize_bounce_page(&fio->encrypted_page); + folio_end_writeback(folio); + } f2fs_bug_on(fio->sbi, !is_set_ckpt_flags(fio->sbi, CP_ERROR_FLAG)); goto out; @@ -4118,7 +4127,10 @@ static void do_write_page(struct f2fs_summary *sum, struct f2fs_io_info *fio) f2fs_invalidate_internal_cache(fio->sbi, fio->old_blkaddr, 1); /* writeout dirty page into bdev */ - f2fs_submit_page_write(fio); + if (fio->is_cache) + f2fs_submit_cache_write(fio); + else + f2fs_submit_page_write(fio); f2fs_update_device_state(fio->sbi, fio->ino, fio->new_blkaddr, 1); out: @@ -4351,14 +4363,13 @@ void f2fs_replace_block(struct f2fs_sb_info *sbi, struct dnode_of_data *dn, f2fs_update_data_blkaddr(dn, new_addr); } -void f2fs_folio_wait_writeback(struct folio *folio, enum page_type type, - bool ordered, bool locked) +void f2fs_folio_wait_writeback(struct folio *folio, bool ordered, bool locked) { if (folio_test_writeback(folio)) { struct f2fs_sb_info *sbi = F2FS_F_SB(folio); /* submit cached LFS IO */ - f2fs_submit_merged_write_folio(sbi, folio, type); + f2fs_submit_merged_write_folio(sbi, folio); /* submit cached IPU IO */ f2fs_submit_merged_ipu_write(sbi, NULL, folio); if (ordered) { diff --git a/fs/f2fs/segment.h b/fs/f2fs/segment.h index fff35c63a00f..341939478fd4 100644 --- a/fs/f2fs/segment.h +++ b/fs/f2fs/segment.h @@ -1014,26 +1014,6 @@ static inline long adjust_flush_cache_number(struct f2fs_sb_info *sbi, int type) return nr_to_write; } -/* - * When writing pages, it'd better align nr_to_write for segment size. - */ -static inline long nr_pages_to_write(struct f2fs_sb_info *sbi, int type, - struct writeback_control *wbc) -{ - long nr_to_write, desired; - - if (wbc->sync_mode != WB_SYNC_NONE) - return 0; - - nr_to_write = wbc->nr_to_write; - desired = BIO_MAX_VECS; - if (type == NODE) - desired <<= 1; - - wbc->nr_to_write = desired; - return desired - nr_to_write; -} - static inline void wake_up_discard_thread(struct f2fs_sb_info *sbi, bool force) { struct discard_cmd_control *dcc = SM_I(sbi)->dcc_info; diff --git a/fs/f2fs/shrinker.c b/fs/f2fs/shrinker.c index 1755c85849e4..20b3fe1f8c07 100644 --- a/fs/f2fs/shrinker.c +++ b/fs/f2fs/shrinker.c @@ -39,7 +39,8 @@ static unsigned long __count_extent_cache(struct f2fs_sb_info *sbi, static unsigned long __count_cache(struct f2fs_sb_info *sbi) { - return sbi->meta_blocks.num_entries; + return sbi->meta_blocks.num_entries + + sbi->node_blocks.num_entries; } unsigned long f2fs_shrink_count(struct shrinker *shrink, diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index 83496c46c89f..bc1cff6cad0d 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -1831,20 +1831,8 @@ static struct inode *f2fs_alloc_inode(struct super_block *sb) static int f2fs_drop_inode(struct inode *inode) { - struct f2fs_sb_info *sbi = F2FS_I_SB(inode); int ret; - /* - * during filesystem shutdown, if checkpoint is disabled, - * drop useless meta/node dirty pages. - */ - if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) { - if (inode->i_ino == F2FS_NODE_INO(sbi)) { - trace_f2fs_drop_inode(inode, 1); - return 1; - } - } - /* * This is to avoid a deadlock condition like below. * writeback_single_inode(inode) @@ -1865,7 +1853,7 @@ static int f2fs_drop_inode(struct inode *inode) f2fs_i_size_write(inode, 0); f2fs_submit_merged_write_cond(F2FS_I_SB(inode), - inode, NULL, 0, DATA); + inode, NULL); truncate_inode_pages_final(inode->i_mapping); if (F2FS_HAS_BLOCKS(inode)) @@ -1939,11 +1927,6 @@ void f2fs_inode_synced(struct inode *inode) */ static void f2fs_dirty_inode(struct inode *inode, int flags) { - struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - - if (inode->i_ino == F2FS_NODE_INO(sbi)) - return; - if (is_inode_flag_set(inode, FI_AUTO_RECOVER)) clear_inode_flag(inode, FI_AUTO_RECOVER); @@ -2039,7 +2022,7 @@ static void f2fs_put_super(struct super_block *sb) if (err || f2fs_cp_error(sbi) || unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) { - truncate_inode_pages_final(NODE_MAPPING(sbi)); + f2fs_truncate_node_caches(sbi, 0, ULONG_MAX); f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); } @@ -2047,11 +2030,8 @@ static void f2fs_put_super(struct super_block *sb) f2fs_destroy_compress_inode(sbi); - iput(sbi->node_inode); - sbi->node_inode = NULL; - - f2fs_destroy_cache(META_CACHE(sbi)); f2fs_destroy_cache(NODE_CACHE(sbi)); + f2fs_destroy_cache(META_CACHE(sbi)); /* Should check the page counts after dropping all node/meta pages */ for (i = 0; i < NR_COUNT_TYPE; i++) { @@ -4379,7 +4359,6 @@ static void init_sb_info(struct f2fs_sb_info *sbi) sbi->allocate_section_hint = le32_to_cpu(raw_super->section_count); sbi->allocate_section_policy = ALLOCATE_FORWARD_NOHINT; F2FS_ROOT_INO(sbi) = le32_to_cpu(raw_super->root_ino); - F2FS_NODE_INO(sbi) = le32_to_cpu(raw_super->node_ino); sbi->cur_victim_sec = NULL_SECNO; sbi->gc_mode = GC_NORMAL; sbi->next_victim_seg[BG_GC] = NULL_SEGNO; @@ -5030,7 +5009,7 @@ static void f2fs_restore_device_alias(struct f2fs_sb_info *sbi) { struct inode *root = d_inode(sbi->sb->s_root); struct f2fs_dir_entry *de; - struct folio *folio; + void *dentry_block = NULL; int i; if (!f2fs_sb_has_device_alias(sbi)) @@ -5045,7 +5024,7 @@ static void f2fs_restore_device_alias(struct f2fs_sb_info *sbi) qstr.name = name; qstr.len = strlen(name); - de = f2fs_find_entry(root, &qstr, &folio); + de = f2fs_find_entry(root, &qstr, &dentry_block); if (!de) continue; @@ -5055,7 +5034,7 @@ static void f2fs_restore_device_alias(struct f2fs_sb_info *sbi) FDEV(i).has_alias = true; iput(inode); } - f2fs_folio_put(folio, 0); + f2fs_put_dentry_block(dentry_block, false); } } @@ -5322,33 +5301,25 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) if (err) goto free_nm; - /* get an inode for node space */ - sbi->node_inode = f2fs_iget(sb, F2FS_NODE_INO(sbi)); - if (IS_ERR(sbi->node_inode)) { - f2fs_err(sbi, "Failed to read node inode"); - err = PTR_ERR(sbi->node_inode); - goto free_stats; - } - /* read root inode and dentry */ root = f2fs_iget(sb, F2FS_ROOT_INO(sbi)); if (IS_ERR(root)) { f2fs_err(sbi, "Failed to read root inode"); err = PTR_ERR(root); - goto free_node_inode; + goto free_ino_entry; } if (!S_ISDIR(root->i_mode) || !root->i_blocks || !root->i_size || !root->i_nlink) { iput(root); err = -EINVAL; - goto free_node_inode; + goto free_ino_entry; } generic_set_sb_d_ops(sb); sb->s_root = d_make_root(root); /* allocate root dentry */ if (!sb->s_root) { err = -ENOMEM; - goto free_node_inode; + goto free_ino_entry; } err = f2fs_init_compress_inode(sbi); @@ -5519,7 +5490,7 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) * Some dirty meta pages can be produced by f2fs_recover_orphan_inodes() * failed by EIO. Then, iput(node_inode) can trigger balance_fs_bg() * followed by f2fs_write_checkpoint() through f2fs_write_node_pages(), which - * falls into an infinite loop in f2fs_sync_meta_pages(). + * falls into an infinite loop in f2fs_sync_meta_caches(). */ f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); /* evict some inodes being cached by GC */ @@ -5530,12 +5501,9 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) free_root_inode: dput(sb->s_root); sb->s_root = NULL; -free_node_inode: +free_ino_entry: f2fs_release_ino_entry(sbi, true); - truncate_inode_pages_final(NODE_MAPPING(sbi)); - iput(sbi->node_inode); - sbi->node_inode = NULL; -free_stats: + f2fs_truncate_node_caches(sbi, 0, ULONG_MAX); f2fs_destroy_stats(sbi); free_nm: /* stop discard thread before destroying node manager */ diff --git a/fs/f2fs/xattr.c b/fs/f2fs/xattr.c index 6728d1488cad..3169e361ea68 100644 --- a/fs/f2fs/xattr.c +++ b/fs/f2fs/xattr.c @@ -138,7 +138,7 @@ static int f2fs_xattr_advise_set(const struct xattr_handler *handler, #ifdef CONFIG_F2FS_FS_SECURITY static int f2fs_initxattrs(struct inode *inode, const struct xattr *xattr_array, - void *folio) + void *fs_data) { const struct xattr *xattr; int err = 0; @@ -146,7 +146,7 @@ static int f2fs_initxattrs(struct inode *inode, const struct xattr *xattr_array, for (xattr = xattr_array; xattr->name != NULL; xattr++) { err = f2fs_setxattr(inode, F2FS_XATTR_INDEX_SECURITY, xattr->name, xattr->value, - xattr->value_len, folio, 0); + xattr->value_len, fs_data, 0); if (err < 0) break; } @@ -154,10 +154,10 @@ static int f2fs_initxattrs(struct inode *inode, const struct xattr *xattr_array, } int f2fs_init_security(struct inode *inode, struct inode *dir, - const struct qstr *qstr, struct folio *ifolio) + const struct qstr *qstr, struct f2fs_cached_block *ientry) { return security_inode_init_security(inode, dir, qstr, - f2fs_initxattrs, ifolio); + f2fs_initxattrs, ientry); } #endif @@ -273,25 +273,25 @@ static struct f2fs_xattr_entry *__find_inline_xattr(struct inode *inode, return entry; } -static int read_inline_xattr(struct inode *inode, struct folio *ifolio, +static int read_inline_xattr(struct inode *inode, struct f2fs_cached_block *ientry, void *txattr_addr) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); unsigned int inline_size = inline_xattr_size(inode); - struct folio *folio = NULL; + struct f2fs_cached_block *in_entry = NULL; void *inline_addr; - if (ifolio) { - inline_addr = inline_xattr_addr(inode, ifolio); + if (ientry) { + inline_addr = inline_xattr_addr(inode, ientry); } else { - folio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(folio)) - return PTR_ERR(folio); + in_entry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(in_entry)) + return PTR_ERR(in_entry); - inline_addr = inline_xattr_addr(inode, folio); + inline_addr = inline_xattr_addr(inode, in_entry); } memcpy(txattr_addr, inline_addr, inline_size); - f2fs_folio_put(folio, true); + f2fs_put_cache(in_entry, true); return 0; } @@ -301,22 +301,22 @@ static int read_xattr_block(struct inode *inode, void *txattr_addr) struct f2fs_sb_info *sbi = F2FS_I_SB(inode); nid_t xnid = F2FS_I(inode)->i_xattr_nid; unsigned int inline_size = inline_xattr_size(inode); - struct folio *xfolio; + struct f2fs_cached_block *xentry; void *xattr_addr; /* The inode already has an extended attribute block. */ - xfolio = f2fs_get_xnode_folio(sbi, xnid); - if (IS_ERR(xfolio)) - return PTR_ERR(xfolio); + xentry = f2fs_get_xnode_cache(sbi, xnid); + if (IS_ERR(xentry)) + return PTR_ERR(xentry); - xattr_addr = folio_address(xfolio); + xattr_addr = cache_address(xentry); memcpy(txattr_addr + inline_size, xattr_addr, VALID_XATTR_BLOCK_SIZE); - f2fs_folio_put(xfolio, true); + f2fs_put_cache(xentry, true); return 0; } -static int lookup_all_xattrs(struct inode *inode, struct folio *ifolio, +static int lookup_all_xattrs(struct inode *inode, struct f2fs_cached_block *ientry, unsigned int index, unsigned int len, const char *name, struct f2fs_xattr_entry **xe, void **base_addr, int *base_size, @@ -340,7 +340,7 @@ static int lookup_all_xattrs(struct inode *inode, struct folio *ifolio, /* read from inline xattr */ if (inline_size) { - err = read_inline_xattr(inode, ifolio, txattr_addr); + err = read_inline_xattr(inode, ientry, txattr_addr); if (err) goto out; @@ -388,7 +388,7 @@ static int lookup_all_xattrs(struct inode *inode, struct folio *ifolio, return err; } -static int read_all_xattrs(struct inode *inode, struct folio *ifolio, +static int read_all_xattrs(struct inode *inode, struct f2fs_cached_block *ientry, void **base_addr) { struct f2fs_xattr_header *header; @@ -405,7 +405,7 @@ static int read_all_xattrs(struct inode *inode, struct folio *ifolio, /* read from inline xattr */ if (inline_size) { - err = read_inline_xattr(inode, ifolio, txattr_addr); + err = read_inline_xattr(inode, ientry, txattr_addr); if (err) goto fail; } @@ -432,14 +432,14 @@ static int read_all_xattrs(struct inode *inode, struct folio *ifolio, } static inline int write_all_xattrs(struct inode *inode, __u32 hsize, - void *txattr_addr, struct folio *ifolio) + void *txattr_addr, struct f2fs_cached_block *ientry) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); size_t inline_size = inline_xattr_size(inode); - struct folio *in_folio = NULL; + struct f2fs_cached_block *in_entry = NULL; void *xattr_addr; void *inline_addr = NULL; - struct folio *xfolio; + struct f2fs_cached_block *xentry; nid_t new_nid = 0; int err = 0; @@ -449,75 +449,74 @@ static inline int write_all_xattrs(struct inode *inode, __u32 hsize, /* write to inline xattr */ if (inline_size) { - if (ifolio) { - inline_addr = inline_xattr_addr(inode, ifolio); + if (ientry) { + inline_addr = inline_xattr_addr(inode, ientry); } else { - in_folio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(in_folio)) { + in_entry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(in_entry)) { f2fs_alloc_nid_failed(sbi, new_nid); - return PTR_ERR(in_folio); + return PTR_ERR(in_entry); } - inline_addr = inline_xattr_addr(inode, in_folio); + inline_addr = inline_xattr_addr(inode, in_entry); } - f2fs_folio_wait_writeback(ifolio ? ifolio : in_folio, - NODE, true, true); + f2fs_cache_wait_writeback(ientry ? ientry : in_entry); /* no need to use xattr node block */ if (hsize <= inline_size) { err = f2fs_truncate_xattr_node(inode); f2fs_alloc_nid_failed(sbi, new_nid); if (err) { - f2fs_folio_put(in_folio, true); + f2fs_put_cache(in_entry, true); return err; } memcpy(inline_addr, txattr_addr, inline_size); - folio_mark_dirty(ifolio ? ifolio : in_folio); + f2fs_mark_cache_dirty(ientry ? ientry : in_entry); goto in_page_out; } } /* write to xattr node block */ if (F2FS_I(inode)->i_xattr_nid) { - xfolio = f2fs_get_xnode_folio(sbi, F2FS_I(inode)->i_xattr_nid); - if (IS_ERR(xfolio)) { - err = PTR_ERR(xfolio); + xentry = f2fs_get_xnode_cache(sbi, F2FS_I(inode)->i_xattr_nid); + if (IS_ERR(xentry)) { + err = PTR_ERR(xentry); f2fs_alloc_nid_failed(sbi, new_nid); goto in_page_out; } f2fs_bug_on(sbi, new_nid); - f2fs_folio_wait_writeback(xfolio, NODE, true, true); + f2fs_cache_wait_writeback(xentry); } else { struct dnode_of_data dn; set_new_dnode(&dn, inode, NULL, NULL, new_nid); - xfolio = f2fs_new_node_folio(&dn, XATTR_NODE_OFFSET); - if (IS_ERR(xfolio)) { - err = PTR_ERR(xfolio); + xentry = f2fs_new_node_cache(&dn, XATTR_NODE_OFFSET); + if (IS_ERR(xentry)) { + err = PTR_ERR(xentry); f2fs_alloc_nid_failed(sbi, new_nid); goto in_page_out; } f2fs_alloc_nid_done(sbi, new_nid); } - xattr_addr = folio_address(xfolio); + xattr_addr = cache_address(xentry); if (inline_size) memcpy(inline_addr, txattr_addr, inline_size); memcpy(xattr_addr, txattr_addr + inline_size, VALID_XATTR_BLOCK_SIZE); if (inline_size) - folio_mark_dirty(ifolio ? ifolio : in_folio); - folio_mark_dirty(xfolio); + f2fs_mark_cache_dirty(ientry ? ientry : in_entry); + f2fs_mark_cache_dirty(xentry); - f2fs_folio_put(xfolio, true); + f2fs_put_cache(xentry, true); in_page_out: - f2fs_folio_put(in_folio, true); + f2fs_put_cache(in_entry, true); return err; } int f2fs_getxattr(struct inode *inode, int index, const char *name, - void *buffer, size_t buffer_size, struct folio *ifolio) + void *buffer, size_t buffer_size, struct f2fs_cached_block *ientry) { - struct f2fs_xattr_entry *entry = NULL; + struct f2fs_xattr_entry *xe = NULL; int error; unsigned int size, len; void *base_addr = NULL; @@ -531,16 +530,16 @@ int f2fs_getxattr(struct inode *inode, int index, const char *name, if (len > F2FS_NAME_LEN) return -ERANGE; - if (!ifolio) + if (!ientry) f2fs_down_read(&F2FS_I(inode)->i_xattr_sem); - error = lookup_all_xattrs(inode, ifolio, index, len, name, - &entry, &base_addr, &base_size, &is_inline); - if (!ifolio) + error = lookup_all_xattrs(inode, ientry, index, len, name, + &xe, &base_addr, &base_size, &is_inline); + if (!ientry) f2fs_up_read(&F2FS_I(inode)->i_xattr_sem); if (error) return error; - size = le16_to_cpu(entry->e_value_size); + size = le16_to_cpu(xe->e_value_size); if (buffer && size > buffer_size) { error = -ERANGE; @@ -548,7 +547,7 @@ int f2fs_getxattr(struct inode *inode, int index, const char *name, } if (buffer) { - char *pval = entry->e_name + entry->e_name_len; + char *pval = xe->e_name + xe->e_name_len; if (base_size - (pval - (char *)base_addr) < size) { error = -ERANGE; @@ -632,7 +631,7 @@ static bool f2fs_xattr_value_same(struct f2fs_xattr_entry *entry, static int __f2fs_setxattr(struct inode *inode, int index, const char *name, const void *value, size_t size, - struct folio *ifolio, int flags) + struct f2fs_cached_block *ientry, int flags) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct f2fs_xattr_entry *here, *last; @@ -656,7 +655,7 @@ static int __f2fs_setxattr(struct inode *inode, int index, if (size > MAX_VALUE_LEN(inode)) return -E2BIG; retry: - error = read_all_xattrs(inode, ifolio, &base_addr); + error = read_all_xattrs(inode, ientry, &base_addr); if (error) return error; @@ -773,7 +772,7 @@ static int __f2fs_setxattr(struct inode *inode, int index, *(u32 *)((u8 *)last + newsize) = 0; } - error = write_all_xattrs(inode, new_hsize, base_addr, ifolio); + error = write_all_xattrs(inode, new_hsize, base_addr, ientry); if (error) goto exit; @@ -807,7 +806,7 @@ static int __f2fs_setxattr(struct inode *inode, int index, int f2fs_setxattr(struct inode *inode, int index, const char *name, const void *value, size_t size, - struct folio *ifolio, int flags) + struct f2fs_cached_block *ientry, int flags) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct f2fs_lock_context lc; @@ -823,9 +822,9 @@ int f2fs_setxattr(struct inode *inode, int index, const char *name, return err; /* this case is only from f2fs_init_inode_metadata */ - if (ifolio) + if (ientry) return __f2fs_setxattr(inode, index, name, value, - size, ifolio, flags); + size, ientry, flags); f2fs_balance_fs(sbi, true); f2fs_lock_op(sbi, &lc); diff --git a/fs/f2fs/xattr.h b/fs/f2fs/xattr.h index bce3d93e4755..f50ed518c5cb 100644 --- a/fs/f2fs/xattr.h +++ b/fs/f2fs/xattr.h @@ -130,9 +130,9 @@ extern const struct xattr_handler f2fs_xattr_security_handler; extern const struct xattr_handler * const f2fs_xattr_handlers[]; int f2fs_setxattr(struct inode *, int, const char *, const void *, - size_t, struct folio *, int); + size_t, struct f2fs_cached_block *, int); int f2fs_getxattr(struct inode *, int, const char *, void *, - size_t, struct folio *); + size_t, struct f2fs_cached_block *); ssize_t f2fs_listxattr(struct dentry *, char *, size_t); int __init f2fs_init_xattr_cache(void); void f2fs_destroy_xattr_cache(void); @@ -142,13 +142,13 @@ void f2fs_destroy_xattr_cache(void); #define f2fs_listxattr NULL static inline int f2fs_setxattr(struct inode *inode, int index, const char *name, const void *value, size_t size, - struct folio *folio, int flags) + struct f2fs_cached_block *ientry, int flags) { return -EOPNOTSUPP; } static inline int f2fs_getxattr(struct inode *inode, int index, const char *name, void *buffer, - size_t buffer_size, struct folio *dfolio) + size_t buffer_size, struct f2fs_cached_block *ientry) { return -EOPNOTSUPP; } @@ -158,10 +158,10 @@ static inline void f2fs_destroy_xattr_cache(void) { } #ifdef CONFIG_F2FS_FS_SECURITY int f2fs_init_security(struct inode *, struct inode *, - const struct qstr *, struct folio *); + const struct qstr *, struct f2fs_cached_block *); #else static inline int f2fs_init_security(struct inode *inode, struct inode *dir, - const struct qstr *qstr, struct folio *ifolio) + const struct qstr *qstr, struct f2fs_cached_block *ientry) { return 0; } diff --git a/include/linux/f2fs_fs.h b/include/linux/f2fs_fs.h index 0c027d00a1ea..105cfeedea74 100644 --- a/include/linux/f2fs_fs.h +++ b/include/linux/f2fs_fs.h @@ -34,7 +34,6 @@ #define F2FS_RESERVED_NODE_NUM 3 #define F2FS_ROOT_INO(sbi) ((sbi)->root_ino_num) -#define F2FS_NODE_INO(sbi) ((sbi)->node_ino_num) #define F2FS_COMPRESS_INO(sbi) (NM_I(sbi)->max_nid) #define F2FS_MAX_QUOTAS 3 -- 2.49.0 ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [f2fs-dev] [PATCH v1 07/12] f2fs: cache: use node cache @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-kernel, linux-f2fs-devel This patch migrates F2FS node block caching from the fake VFS inode page cache (sbi->node_inode) to node cache (sbi->node_blocks). It updates node related helpers to use node cache APIs and reference to struct f2fs_cached_block, and removes sbi->node_inode, especially, unifies inline and regular dentry block handling via struct f2fs_dentry_block_ref. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/acl.c | 26 +- fs/f2fs/acl.h | 8 +- fs/f2fs/cache.c | 7 +- fs/f2fs/cache.h | 1 + fs/f2fs/checkpoint.c | 7 +- fs/f2fs/compress.c | 18 +- fs/f2fs/data.c | 236 +++++------ fs/f2fs/debug.c | 12 +- fs/f2fs/dir.c | 168 ++++---- fs/f2fs/extent_cache.c | 14 +- fs/f2fs/f2fs.h | 223 +++++----- fs/f2fs/file.c | 74 ++-- fs/f2fs/gc.c | 44 +- fs/f2fs/inline.c | 279 ++++++------- fs/f2fs/inode.c | 155 ++++--- fs/f2fs/namei.c | 118 +++--- fs/f2fs/node.c | 875 +++++++++++++++++++--------------------- fs/f2fs/node.h | 107 +++-- fs/f2fs/recovery.c | 115 +++--- fs/f2fs/segment.c | 59 +-- fs/f2fs/segment.h | 20 - fs/f2fs/shrinker.c | 3 +- fs/f2fs/super.c | 56 +-- fs/f2fs/xattr.c | 123 +++--- fs/f2fs/xattr.h | 12 +- include/linux/f2fs_fs.h | 1 - 26 files changed, 1329 insertions(+), 1432 deletions(-) diff --git a/fs/f2fs/acl.c b/fs/f2fs/acl.c index d3253549173e..34c9aa279040 100644 --- a/fs/f2fs/acl.c +++ b/fs/f2fs/acl.c @@ -181,7 +181,7 @@ static void *f2fs_acl_to_disk(struct f2fs_sb_info *sbi, } static struct posix_acl *__f2fs_get_acl(struct inode *inode, int type, - struct folio *dfolio) + struct f2fs_cached_block *entry) { int name_index = F2FS_XATTR_INDEX_POSIX_ACL_DEFAULT; void *value = NULL; @@ -191,13 +191,13 @@ static struct posix_acl *__f2fs_get_acl(struct inode *inode, int type, if (type == ACL_TYPE_ACCESS) name_index = F2FS_XATTR_INDEX_POSIX_ACL_ACCESS; - retval = f2fs_getxattr(inode, name_index, "", NULL, 0, dfolio); + retval = f2fs_getxattr(inode, name_index, "", NULL, 0, entry); if (retval > 0) { value = f2fs_kmalloc(F2FS_I_SB(inode), retval, GFP_F2FS_ZERO); if (!value) return ERR_PTR(-ENOMEM); retval = f2fs_getxattr(inode, name_index, "", value, - retval, dfolio); + retval, entry); } if (retval > 0) @@ -242,7 +242,7 @@ static int f2fs_acl_update_mode(struct mnt_idmap *idmap, static int __f2fs_set_acl(struct mnt_idmap *idmap, struct inode *inode, int type, - struct posix_acl *acl, struct folio *ifolio) + struct posix_acl *acl, struct f2fs_cached_block *ientry) { int name_index; void *value = NULL; @@ -253,7 +253,7 @@ static int __f2fs_set_acl(struct mnt_idmap *idmap, switch (type) { case ACL_TYPE_ACCESS: name_index = F2FS_XATTR_INDEX_POSIX_ACL_ACCESS; - if (acl && !ifolio) { + if (acl && !ientry) { error = f2fs_acl_update_mode(idmap, inode, &mode, &acl); if (error) return error; @@ -279,7 +279,7 @@ static int __f2fs_set_acl(struct mnt_idmap *idmap, } } - error = f2fs_setxattr(inode, name_index, "", value, size, ifolio, 0); + error = f2fs_setxattr(inode, name_index, "", value, size, ientry, 0); kfree(value); if (!error) @@ -374,7 +374,7 @@ static int f2fs_acl_create_masq(struct posix_acl *acl, umode_t *mode_p) static int f2fs_acl_create(struct inode *dir, umode_t *mode, struct posix_acl **default_acl, struct posix_acl **acl, - struct folio *dfolio) + struct f2fs_cached_block *entry) { struct posix_acl *p; struct posix_acl *clone; @@ -386,7 +386,7 @@ static int f2fs_acl_create(struct inode *dir, umode_t *mode, if (S_ISLNK(*mode) || !IS_POSIXACL(dir)) return 0; - p = __f2fs_get_acl(dir, ACL_TYPE_DEFAULT, dfolio); + p = __f2fs_get_acl(dir, ACL_TYPE_DEFAULT, entry); if (!p || p == ERR_PTR(-EOPNOTSUPP)) { *mode &= ~current_umask(); return 0; @@ -423,13 +423,13 @@ static int f2fs_acl_create(struct inode *dir, umode_t *mode, return ret; } -int f2fs_init_acl(struct inode *inode, struct inode *dir, struct folio *ifolio, - struct folio *dfolio) +int f2fs_init_acl(struct inode *inode, struct inode *dir, struct f2fs_cached_block *ientry, + struct f2fs_cached_block *dentry) { struct posix_acl *default_acl = NULL, *acl = NULL; int error; - error = f2fs_acl_create(dir, &inode->i_mode, &default_acl, &acl, dfolio); + error = f2fs_acl_create(dir, &inode->i_mode, &default_acl, &acl, dentry); if (error) return error; @@ -437,7 +437,7 @@ int f2fs_init_acl(struct inode *inode, struct inode *dir, struct folio *ifolio, if (default_acl) { error = __f2fs_set_acl(NULL, inode, ACL_TYPE_DEFAULT, - default_acl, ifolio); + default_acl, ientry); posix_acl_release(default_acl); } else { inode->i_default_acl = NULL; @@ -445,7 +445,7 @@ int f2fs_init_acl(struct inode *inode, struct inode *dir, struct folio *ifolio, if (acl) { if (!error) error = __f2fs_set_acl(NULL, inode, ACL_TYPE_ACCESS, - acl, ifolio); + acl, ientry); posix_acl_release(acl); } else { inode->i_acl = NULL; diff --git a/fs/f2fs/acl.h b/fs/f2fs/acl.h index 20e87e63c089..0f639367a0ab 100644 --- a/fs/f2fs/acl.h +++ b/fs/f2fs/acl.h @@ -36,14 +36,16 @@ struct f2fs_acl_header { struct posix_acl *f2fs_get_acl(struct inode *, int, bool); int f2fs_set_acl(struct mnt_idmap *, struct dentry *, struct posix_acl *, int); -int f2fs_init_acl(struct inode *, struct inode *, struct folio *ifolio, - struct folio *dfolio); +int f2fs_init_acl(struct inode *inode, struct inode *dir, + struct f2fs_cached_block *ientry, + struct f2fs_cached_block *dentry); #else #define f2fs_get_acl NULL #define f2fs_set_acl NULL static inline int f2fs_init_acl(struct inode *inode, struct inode *dir, - struct folio *ifolio, struct folio *dfolio) + struct f2fs_cached_block *ientry, + struct f2fs_cached_block *dentry) { return 0; } diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index 6c9b7a6d2313..3176d62ce25b 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -29,7 +29,7 @@ void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, return; /* submit cached bio */ - f2fs_submit_merged_write_cache(entry, type); + f2fs_submit_merged_write_cache(entry->cache->sbi, entry, 0, type); wait_on_bit_io(&entry->state, F2FS_BLOCK_WRITEBACK, TASK_UNINTERRUPTIBLE); @@ -67,8 +67,8 @@ bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry) f2fs_cache_set_uptodate(entry); #ifdef CONFIG_F2FS_CHECK_FS - if (cache->type == F2FS_NODE_CACHE && IS_INODE(cache_folio(entry))) - f2fs_inode_chksum_set(cache->sbi, cache_folio(entry)); + if (f2fs_is_node_cache(entry) && IS_INODE(entry)) + f2fs_inode_chksum_set(cache->sbi, entry); #endif if (f2fs_cache_test_dirty(entry)) @@ -640,6 +640,7 @@ static int f2fs_cache_writeback_kthread(void *data) continue; f2fs_write_meta_caches(sbi); + f2fs_write_node_caches(sbi); } return 0; } diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index d843b9caba47..397019cfb861 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -52,6 +52,7 @@ struct f2fs_cached_block_list { }; #define IS_META_CACHE(cache) (cache->type == F2FS_META_CACHE) +#define IS_NODE_CACHE(cache) (cache->type == F2FS_NODE_CACHE) /* Flags for f2fs_cached_block state */ enum f2fs_cached_state { diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c index 1a7083540b82..508132652693 100644 --- a/fs/f2fs/checkpoint.c +++ b/fs/f2fs/checkpoint.c @@ -1489,10 +1489,6 @@ static bool __need_flush_quota(struct f2fs_sb_info *sbi) */ static int block_operations(struct f2fs_sb_info *sbi) { - struct writeback_control wbc = { - .sync_mode = WB_SYNC_ALL, - .nr_to_write = LONG_MAX, - }; int err = 0, cnt = 0; /* @@ -1556,7 +1552,8 @@ static int block_operations(struct f2fs_sb_info *sbi) if (get_pages(sbi, F2FS_DIRTY_NODES)) { f2fs_up_write(&sbi->node_write); atomic_inc(&sbi->wb_sync_req[NODE]); - err = f2fs_sync_node_pages(sbi, &wbc, false, FS_CP_NODE_IO); + err = f2fs_writeback_node_caches(sbi, LONG_MAX, + true, false, FS_CP_NODE_IO); atomic_dec(&sbi->wb_sync_req[NODE]); if (err) { f2fs_up_write(&sbi->node_change); diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c index bb749f6257a1..676a5559357f 100644 --- a/fs/f2fs/compress.c +++ b/fs/f2fs/compress.c @@ -909,7 +909,7 @@ bool f2fs_sanity_check_cluster(struct dnode_of_data *dn) } for (i = 1, count = 1; i < cluster_size; i++, count++) { - block_t blkaddr = data_blkaddr(dn->inode, dn->node_folio, + block_t blkaddr = data_blkaddr(dn->inode, dn->node_entry, dn->ofs_in_node + i); /* [COMPR_ADDR, ..., COMPR_ADDR] */ @@ -950,7 +950,7 @@ static int __f2fs_get_cluster_blocks(struct inode *inode, int count, i; for (i = 0, count = 0; i < cluster_size; i++) { - block_t blkaddr = data_blkaddr(dn->inode, dn->node_folio, + block_t blkaddr = data_blkaddr(dn->inode, dn->node_entry, dn->ofs_in_node + i); if (__is_valid_data_blkaddr(blkaddr)) @@ -1146,7 +1146,7 @@ static int prepare_compress_overwrite(struct compress_ctx *cc, goto release_and_retry; } - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); f2fs_compress_ctx_add_page(cc, folio); if (!folio_test_uptodate(folio)) { @@ -1324,7 +1324,7 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc, goto out_unlock_op; for (i = 0; i < cc->cluster_size; i++) { - if (data_blkaddr(dn.inode, dn.node_folio, + if (data_blkaddr(dn.inode, dn.node_entry, dn.ofs_in_node + i) == NULL_ADDR) goto out_put_dnode; } @@ -1356,7 +1356,7 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc, page_folio(cc->rpages[i + 1])->index, cic); fio.compressed_page = cc->cpages[i]; - fio.old_blkaddr = data_blkaddr(dn.inode, dn.node_folio, + fio.old_blkaddr = data_blkaddr(dn.inode, dn.node_entry, dn.ofs_in_node + i + 1); /* wait for GCed page writeback via generic cache */ @@ -1567,7 +1567,7 @@ static int f2fs_write_raw_pages(struct compress_ctx *cc, if (folio_test_writeback(folio)) { if (wbc->sync_mode == WB_SYNC_NONE) goto continue_unlock; - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); } if (!folio_clear_dirty_for_io(folio)) @@ -1899,14 +1899,14 @@ void f2fs_put_folio_dic(struct folio *folio, bool in_task) unsigned int f2fs_cluster_blocks_are_contiguous(struct dnode_of_data *dn, unsigned int ofs_in_node) { - bool compressed = data_blkaddr(dn->inode, dn->node_folio, + bool compressed = data_blkaddr(dn->inode, dn->node_entry, ofs_in_node) == COMPRESS_ADDR; int i = compressed ? 1 : 0; - block_t first_blkaddr = data_blkaddr(dn->inode, dn->node_folio, + block_t first_blkaddr = data_blkaddr(dn->inode, dn->node_entry, ofs_in_node + i); for (i += 1; i < F2FS_I(dn->inode)->i_cluster_size; i++) { - block_t blkaddr = data_blkaddr(dn->inode, dn->node_folio, + block_t blkaddr = data_blkaddr(dn->inode, dn->node_entry, ofs_in_node + i); if (!__is_valid_data_blkaddr(blkaddr)) diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index 92c3293f0a1e..af05a1f6a00f 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -58,15 +58,13 @@ bool f2fs_is_cp_guaranteed(const struct folio *folio) { struct address_space *mapping = folio->mapping; struct inode *inode; - struct f2fs_sb_info *sbi; if (fscrypt_is_bounce_folio(folio)) return folio_test_f2fs_gcing(fscrypt_pagecache_folio(folio)); inode = mapping->host; - if (inode->i_ino == F2FS_NODE_INO(sbi) || - S_ISDIR(inode->i_mode)) + if (S_ISDIR(inode->i_mode)) return true; if ((S_ISREG(inode->i_mode) && IS_NOQUOTA(inode)) || @@ -75,20 +73,6 @@ bool f2fs_is_cp_guaranteed(const struct folio *folio) return false; } -static enum count_type __read_io_type(struct folio *folio) -{ - struct address_space *mapping = folio->mapping; - - if (mapping) { - struct inode *inode = mapping->host; - struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - - if (inode->i_ino == F2FS_NODE_INO(sbi)) - return F2FS_RD_NODE; - } - return F2FS_RD_DATA; -} - /* postprocessing steps for read bios */ enum bio_post_read_step { #ifdef CONFIG_FS_ENCRYPTION @@ -167,13 +151,7 @@ static void f2fs_finish_read_bio(struct bio *bio, bool in_task) } while (nr_pages--) - dec_page_count(F2FS_F_SB(folio), __read_io_type(folio)); - - if (bio->bi_status == BLK_STS_OK && - F2FS_F_SB(folio)->node_inode && is_node_folio(folio) && - f2fs_sanity_check_node_footer(F2FS_F_SB(folio), - folio, folio->index, NODE_TYPE_REGULAR, true)) - bio->bi_status = BLK_STS_IOERR; + dec_page_count(F2FS_F_SB(folio), F2FS_RD_DATA); if (finished) folio_end_read(folio, bio->bi_status == BLK_STS_OK); @@ -376,14 +354,6 @@ static void f2fs_write_end_bio(struct bio *bio) } } - if (is_node_folio(folio)) { - f2fs_sanity_check_node_footer(sbi, folio, - folio->index, NODE_TYPE_REGULAR, true); - f2fs_bug_on(sbi, folio->index != nid_of_node(folio)); - } - if (f2fs_in_warm_node_list(folio)) - f2fs_del_fsync_node_entry(sbi, folio); - dec_page_count(sbi, type); /* @@ -444,6 +414,12 @@ static void f2fs_cache_read_end_io(struct bio *bio) next = entry->next_entry; entry->next_entry = NULL; + if (bio->bi_status == BLK_STS_OK && + f2fs_is_node_cache(entry) && + f2fs_sanity_check_node_footer(sbi, entry, + entry->index, NODE_TYPE_REGULAR, true)) + bio->bi_status = BLK_STS_IOERR; + if (bio->bi_status == BLK_STS_OK) f2fs_cache_set_uptodate(entry); @@ -474,6 +450,14 @@ static void f2fs_cache_write_end_io(struct bio *bio) next = entry->next_entry; entry->next_entry = NULL; + if (f2fs_is_node_cache(entry)) { + f2fs_sanity_check_node_footer(sbi, entry, + entry->index, NODE_TYPE_REGULAR, true); + f2fs_bug_on(sbi, entry->index != nid_of_node(entry)); + } + if (f2fs_in_warm_node_list(entry)) + f2fs_del_fsync_node_entry(sbi, entry); + dec_page_count(sbi, F2FS_WB_CP_DATA); if (!get_pages(sbi, F2FS_WB_CP_DATA) && @@ -681,14 +665,14 @@ static void __submit_merged_bio(struct f2fs_bio_info *io) } static bool __has_merged_page(struct bio *bio, struct inode *inode, - struct folio *folio, nid_t ino) + struct folio *folio) { struct folio_iter fi; if (!bio) return false; - if (!inode && !folio && !ino) + if (!inode && !folio) return true; if (f2fs_is_cache_bio(bio)) @@ -712,8 +696,6 @@ static bool __has_merged_page(struct bio *bio, struct inode *inode, return true; if (folio && folio == target) return true; - if (ino && ino == ino_of_node(target)) - return true; } return false; @@ -782,24 +764,23 @@ static void __f2fs_submit_merged_write(struct f2fs_sb_info *sbi, static void __submit_merged_write_cond(struct f2fs_sb_info *sbi, struct inode *inode, struct folio *folio, - nid_t ino, enum page_type type, bool writeback) + bool writeback) { enum temp_type temp; bool ret = true; - bool force = !inode && !folio && !ino; + bool force = !inode && !folio; for (temp = HOT; temp < NR_TEMP_TYPE; temp++) { - if (!force) { - enum page_type btype = PAGE_TYPE_OF_BIO(type); - struct f2fs_bio_info *io = sbi->write_io[btype] + temp; + if (!force) { + struct f2fs_bio_info *io = sbi->write_io[DATA] + temp; struct f2fs_lock_context lc; f2fs_down_read_trace(&io->io_rwsem, &lc); - ret = __has_merged_page(io->bio, inode, folio, ino); + ret = __has_merged_page(io->bio, inode, folio); f2fs_up_read_trace(&io->io_rwsem, &lc); } if (ret) { - __f2fs_submit_merged_write(sbi, type, temp); + __f2fs_submit_merged_write(sbi, DATA, temp); /* * For waitting writebck case, if the bio owned by the * folio is already submitted, we do not need to submit @@ -808,33 +789,23 @@ static void __submit_merged_write_cond(struct f2fs_sb_info *sbi, if (writeback) break; } - - /* TODO: use HOT temp only for meta pages now. */ - if (type >= META) - break; } } -void f2fs_submit_merged_write(struct f2fs_sb_info *sbi, enum page_type type) -{ - __submit_merged_write_cond(sbi, NULL, NULL, 0, type, false); -} - void f2fs_submit_merged_write_cond(struct f2fs_sb_info *sbi, - struct inode *inode, struct folio *folio, - nid_t ino, enum page_type type) + struct inode *inode, struct folio *folio) { - __submit_merged_write_cond(sbi, inode, folio, ino, type, false); + __submit_merged_write_cond(sbi, inode, folio, false); } void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, - struct folio *folio, enum page_type type) + struct folio *folio) { - __submit_merged_write_cond(sbi, NULL, folio, 0, type, true); + __submit_merged_write_cond(sbi, NULL, folio, true); } static bool __has_merged_cache(struct bio *bio, - struct f2fs_cached_block *target) + struct f2fs_cached_block *target, nid_t ino) { struct f2fs_cached_block *entry; @@ -846,27 +817,32 @@ static bool __has_merged_cache(struct bio *bio, while (entry) { if (target && entry == target) return true; + if (ino && ino_of_node(entry) == ino) + return true; entry = entry->next_entry; } return false; } -bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, - enum page_type type) +bool f2fs_submit_merged_write_cache(struct f2fs_sb_info *sbi, + struct f2fs_cached_block *entry, + nid_t ino, enum page_type type) { - struct f2fs_sb_info *sbi = entry->cache->sbi; enum temp_type temp; bool ret = false; + bool force = !entry && !ino; for (temp = HOT; temp < NR_TEMP_TYPE; temp++) { enum page_type btype = PAGE_TYPE_OF_BIO(type); struct f2fs_bio_info *io = sbi->write_io[btype] + temp; struct f2fs_lock_context lc; - bool merged; + bool merged = true; - f2fs_down_read_trace(&io->io_rwsem, &lc); - merged = __has_merged_cache(io->bio, entry); - f2fs_up_read_trace(&io->io_rwsem, &lc); + if (!force) { + f2fs_down_read_trace(&io->io_rwsem, &lc); + merged = __has_merged_cache(io->bio, entry, ino); + f2fs_up_read_trace(&io->io_rwsem, &lc); + } if (merged) { __f2fs_submit_merged_write(sbi, type, temp); @@ -880,6 +856,14 @@ bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, return ret; } +void f2fs_submit_merged_write(struct f2fs_sb_info *sbi, enum page_type type) +{ + if (type == DATA) + __submit_merged_write_cond(sbi, NULL, NULL, false); + else + f2fs_submit_merged_write_cache(sbi, NULL, 0, type); +} + void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi) { f2fs_submit_merged_write(sbi, DATA); @@ -916,7 +900,7 @@ int f2fs_submit_page_bio(struct f2fs_io_info *fio) wbc_account_cgroup_owner(fio->io_wbc, fio_folio, PAGE_SIZE); inc_page_count(fio->sbi, is_read_io(fio->op) ? - __read_io_type(data_folio) : WB_DATA_TYPE(fio->folio, false)); + F2FS_RD_DATA : WB_DATA_TYPE(fio->folio, false)); if (is_read_io(bio_op(bio))) f2fs_submit_read_bio(fio->sbi, bio, fio->type); @@ -1052,8 +1036,8 @@ void f2fs_submit_merged_ipu_write(struct f2fs_sb_info *sbi, if (target) found = (target == be->bio); else - found = __has_merged_page(be->bio, NULL, - folio, 0); + found = __has_merged_page(be->bio, + NULL, folio); if (found) break; } @@ -1069,8 +1053,8 @@ void f2fs_submit_merged_ipu_write(struct f2fs_sb_info *sbi, if (target) found = (target == be->bio); else - found = __has_merged_page(be->bio, NULL, - folio, 0); + found = __has_merged_page(be->bio, + NULL, folio); if (found) { target = be->bio; del_bio_entry(be); @@ -1461,7 +1445,7 @@ static void f2fs_submit_page_read(struct inode *inode, struct fsverity_info *vi, static void __set_data_blkaddr(struct dnode_of_data *dn, block_t blkaddr) { - __le32 *addr = get_dnode_addr(dn->inode, dn->node_folio); + __le32 *addr = get_dnode_addr(dn->inode, dn->node_entry); dn->data_blkaddr = blkaddr; addr[dn->ofs_in_node] = cpu_to_le32(dn->data_blkaddr); @@ -1475,9 +1459,9 @@ static void __set_data_blkaddr(struct dnode_of_data *dn, block_t blkaddr) */ void f2fs_set_data_blkaddr(struct dnode_of_data *dn, block_t blkaddr) { - f2fs_folio_wait_writeback(dn->node_folio, NODE, true, true); + f2fs_cache_wait_writeback(dn->node_entry); __set_data_blkaddr(dn, blkaddr); - if (folio_mark_dirty(dn->node_folio)) + if (f2fs_mark_cache_dirty(dn->node_entry)) dn->node_changed = true; } @@ -1505,7 +1489,7 @@ int f2fs_reserve_new_blocks(struct dnode_of_data *dn, blkcnt_t count) trace_f2fs_reserve_new_blocks(dn->inode, dn->nid, dn->ofs_in_node, count); - f2fs_folio_wait_writeback(dn->node_folio, NODE, true, true); + f2fs_cache_wait_writeback(dn->node_entry); for (; count > 0; dn->ofs_in_node++) { block_t blkaddr = f2fs_data_blkaddr(dn); @@ -1516,7 +1500,7 @@ int f2fs_reserve_new_blocks(struct dnode_of_data *dn, blkcnt_t count) } } - if (folio_mark_dirty(dn->node_folio)) + if (f2fs_mark_cache_dirty(dn->node_entry)) dn->node_changed = true; return 0; } @@ -1534,7 +1518,7 @@ int f2fs_reserve_new_block(struct dnode_of_data *dn) int f2fs_reserve_block(struct dnode_of_data *dn, pgoff_t index) { - bool need_put = dn->inode_folio ? false : true; + bool need_put = dn->inode_entry ? false : true; int err; err = f2fs_get_dnode_of_data(dn, index, ALLOC_NODE); @@ -1700,11 +1684,11 @@ struct folio *f2fs_get_lock_data_folio(struct inode *inode, pgoff_t index, * * Also, caller should grab and release a rwsem by calling f2fs_lock_op() and * f2fs_unlock_op(). - * Note that, ifolio is set only by make_empty_dir, and if any error occur, - * ifolio should be released by this function. + * Note that, ientry is set only by make_empty_dir, and if any error occur, + * ientry should be released by this function. */ struct folio *f2fs_get_new_data_folio(struct inode *inode, - struct folio *ifolio, pgoff_t index, bool new_i_size) + struct f2fs_cached_block *ientry, pgoff_t index, bool new_i_size) { struct address_space *mapping = inode->i_mapping; struct folio *folio; @@ -1714,20 +1698,20 @@ struct folio *f2fs_get_new_data_folio(struct inode *inode, folio = f2fs_grab_cache_folio(mapping, index, true); if (IS_ERR(folio)) { /* - * before exiting, we should make sure ifolio will be released + * before exiting, we should make sure ientry will be released * if any error occur. */ - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); return ERR_PTR(-ENOMEM); } - set_new_dnode(&dn, inode, ifolio, NULL, 0); + set_new_dnode(&dn, inode, ientry, NULL, 0); err = f2fs_reserve_block(&dn, index); if (err) { f2fs_folio_put(folio, true); return ERR_PTR(err); } - if (!ifolio) + if (!ientry) f2fs_put_dnode(&dn); if (folio_test_uptodate(folio)) @@ -1740,8 +1724,8 @@ struct folio *f2fs_get_new_data_folio(struct inode *inode, } else { f2fs_folio_put(folio, true); - /* if ifolio exists, blkaddr should be NEW_ADDR */ - f2fs_bug_on(F2FS_I_SB(inode), ifolio); + /* if ientry exists, blkaddr should be NEW_ADDR */ + f2fs_bug_on(F2FS_I_SB(inode), ientry); folio = f2fs_get_lock_data_folio(inode, index, true); if (IS_ERR(folio)) return folio; @@ -1778,7 +1762,7 @@ static int __allocate_data_block(struct dnode_of_data *dn, int seg_type) set_summary(&sum, dn->nid, dn->ofs_in_node, ni.version); old_blkaddr = dn->data_blkaddr; - err = f2fs_allocate_data_block(sbi, NULL, old_blkaddr, + err = f2fs_allocate_data_block(sbi, old_blkaddr, &dn->data_blkaddr, &sum, seg_type, NULL); if (err) { if (old_blkaddr == NULL_ADDR) @@ -1990,7 +1974,7 @@ int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map, int flag) start_pgofs = pgofs; prealloc = 0; last_ofs_in_node = ofs_in_node = dn.ofs_in_node; - end_offset = ADDRS_PER_PAGE(dn.node_folio, inode); + end_offset = ADDRS_PER_PAGE(dn.node_entry, inode); next_block: blkaddr = f2fs_data_blkaddr(&dn); @@ -2241,15 +2225,15 @@ static int f2fs_xattr_fiemap(struct inode *inode, if (f2fs_has_inline_xattr(inode)) { int offset; - struct folio *folio = f2fs_grab_cache_folio(NODE_MAPPING(sbi), - inode->i_ino, false); + struct f2fs_cached_block *entry = + f2fs_grab_node_cache(sbi, inode->i_ino); - if (IS_ERR(folio)) - return PTR_ERR(folio); + if (IS_ERR(entry)) + return PTR_ERR(entry); err = f2fs_get_node_info(sbi, inode->i_ino, &ni, false); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); return err; } @@ -2261,7 +2245,7 @@ static int f2fs_xattr_fiemap(struct inode *inode, phys += offset; len = inline_xattr_size(inode); - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); flags = FIEMAP_EXTENT_DATA_INLINE | FIEMAP_EXTENT_NOT_ALIGNED; @@ -2275,22 +2259,22 @@ static int f2fs_xattr_fiemap(struct inode *inode, } if (xnid) { - struct folio *folio = f2fs_grab_cache_folio(NODE_MAPPING(sbi), - xnid, false); + struct f2fs_cached_block *entry = + f2fs_grab_node_cache(sbi, xnid); - if (IS_ERR(folio)) - return PTR_ERR(folio); + if (IS_ERR(entry)) + return PTR_ERR(entry); err = f2fs_get_node_info(sbi, xnid, &ni, false); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); return err; } phys = F2FS_BLK_TO_BYTES(ni.blk_addr); len = inode->i_sb->s_blocksize; - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); flags = FIEMAP_EXTENT_LAST; } @@ -2649,7 +2633,7 @@ int f2fs_read_multi_pages(struct compress_ctx *cc, struct bio **bio_ret, for (i = 1; i < cc->cluster_size; i++) { block_t blkaddr; - blkaddr = from_dnode ? data_blkaddr(dn.inode, dn.node_folio, + blkaddr = from_dnode ? data_blkaddr(dn.inode, dn.node_entry, dn.ofs_in_node + i) : ei.blk + i - 1; @@ -2683,7 +2667,7 @@ int f2fs_read_multi_pages(struct compress_ctx *cc, struct bio **bio_ret, block_t blkaddr; struct bio_post_read_ctx *ctx; - blkaddr = from_dnode ? data_blkaddr(dn.inode, dn.node_folio, + blkaddr = from_dnode ? data_blkaddr(dn.inode, dn.node_entry, dn.ofs_in_node + i + 1) : ei.blk + i; @@ -3714,7 +3698,7 @@ static int f2fs_write_cache_pages(struct address_space *mapping, if (folio_test_writeback(folio)) { if (wbc->sync_mode == WB_SYNC_NONE) goto continue_unlock; - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); } if (!folio_clear_dirty_for_io(folio)) @@ -3796,8 +3780,8 @@ static int f2fs_write_cache_pages(struct address_space *mapping, mapping->writeback_index = done_index; if (nwritten) - f2fs_submit_merged_write_cond(F2FS_M_SB(mapping), mapping->host, - NULL, 0, DATA); + f2fs_submit_merged_write_cond(F2FS_M_SB(mapping), + mapping->host, NULL); /* submit cached bio of IPU write */ if (bio) f2fs_submit_merged_ipu_write(sbi, &bio, NULL); @@ -3975,7 +3959,7 @@ static int prepare_write_begin(struct f2fs_sb_info *sbi, pgoff_t index = folio->index; struct dnode_of_data dn; struct f2fs_lock_context lc; - struct folio *ifolio; + struct f2fs_cached_block *ientry; bool locked = false; int flag = F2FS_GET_BLOCK_PRE_AIO; int err = 0; @@ -4005,20 +3989,20 @@ static int prepare_write_begin(struct f2fs_sb_info *sbi, restart: /* check inline_data */ - ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(ifolio)) { - err = PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(ientry)) { + err = PTR_ERR(ientry); goto unlock_out; } - set_new_dnode(&dn, inode, ifolio, ifolio, 0); + set_new_dnode(&dn, inode, ientry, ientry, 0); if (f2fs_has_inline_data(inode)) { if (pos + len <= MAX_INLINE_DATA(inode)) { - f2fs_do_read_inline_data(folio, ifolio); + f2fs_do_read_inline_data(folio, ientry); set_inode_flag(inode, FI_DATA_EXIST); if (inode->i_nlink) - folio_set_f2fs_inline(ifolio); + f2fs_cache_set_inline(ientry); goto out; } err = f2fs_convert_inline_folio(&dn, folio); @@ -4065,14 +4049,14 @@ static int __find_data_block(struct inode *inode, pgoff_t index, block_t *blk_addr) { struct dnode_of_data dn; - struct folio *ifolio; + struct f2fs_cached_block *ientry; int err = 0; - ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); - set_new_dnode(&dn, inode, ifolio, ifolio, 0); + set_new_dnode(&dn, inode, ientry, ientry, 0); if (!f2fs_lookup_read_extent_cache_block(inode, index, &dn.data_blkaddr)) { @@ -4094,17 +4078,17 @@ static int __reserve_data_block(struct inode *inode, pgoff_t index, struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct dnode_of_data dn; struct f2fs_lock_context lc; - struct folio *ifolio; + struct f2fs_cached_block *ientry; int err = 0; f2fs_map_lock(sbi, &lc, F2FS_GET_BLOCK_PRE_AIO); - ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(ifolio)) { - err = PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(ientry)) { + err = PTR_ERR(ientry); goto unlock_out; } - set_new_dnode(&dn, inode, ifolio, ifolio, 0); + set_new_dnode(&dn, inode, ientry, ientry, 0); if (!f2fs_lookup_read_extent_cache_block(dn.inode, index, &dn.data_blkaddr)) @@ -4260,7 +4244,7 @@ static int f2fs_write_begin(const struct kiocb *iocb, } } - f2fs_folio_wait_writeback(folio, DATA, false, true); + f2fs_folio_wait_writeback(folio, false, true); if (len == folio_size(folio) || folio_test_uptodate(folio)) return 0; @@ -4377,12 +4361,8 @@ void f2fs_invalidate_folio(struct folio *folio, size_t offset, size_t length) return; if (folio_test_dirty(folio)) { - if (inode->i_ino == F2FS_NODE_INO(sbi)) { - dec_page_count(sbi, F2FS_DIRTY_NODES); - } else { - inode_dec_dirty_pages(inode); - f2fs_remove_dirty_inode(inode); - } + inode_dec_dirty_pages(inode); + f2fs_remove_dirty_inode(inode); } if (offset || length != folio_size(folio)) diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c index 14059a50739c..bedaade92677 100644 --- a/fs/f2fs/debug.c +++ b/fs/f2fs/debug.c @@ -222,8 +222,7 @@ static void update_general_status(struct f2fs_sb_info *sbi) si->free_secs = free_sections(sbi); si->prefree_count = prefree_segments(sbi); si->dirty_count = dirty_segments(sbi); - if (sbi->node_inode) - si->node_pages = NODE_MAPPING(sbi)->nrpages; + si->node_caches = NODE_CACHE(sbi)->num_entries; si->meta_caches = META_CACHE(sbi)->num_entries; #ifdef CONFIG_F2FS_FS_COMPRESSION if (sbi->compress_inode) { @@ -382,13 +381,10 @@ static void update_mem_info(struct f2fs_sb_info *sbi) } si->page_mem = 0; - if (sbi->node_inode) { - unsigned long npages = NODE_MAPPING(sbi)->nrpages; - - si->page_mem += (unsigned long long)npages << PAGE_SHIFT; - } si->page_mem += (unsigned long long)META_CACHE(sbi)->num_entries << PAGE_SHIFT; si->cache_mem += META_CACHE(sbi)->num_entries * sizeof(struct f2fs_cached_block); + si->page_mem += (unsigned long long)NODE_CACHE(sbi)->num_entries << PAGE_SHIFT; + si->cache_mem += NODE_CACHE(sbi)->num_entries * sizeof(struct f2fs_cached_block); #ifdef CONFIG_F2FS_FS_COMPRESSION if (sbi->compress_inode) { unsigned long npages = COMPRESS_MAPPING(sbi)->nrpages; @@ -696,7 +692,7 @@ static int stat_show(struct seq_file *s, void *v) si->aw_cnt, si->max_aw_cnt); seq_printf(s, " - compress: %4d, hit:%8d\n", si->compress_pages, si->compress_page_hit); seq_printf(s, " - nodes: %4d in %4d\n", - si->ndirty_node, si->node_pages); + si->ndirty_node, si->node_caches); seq_printf(s, " - dents: %4d in dirs:%4d (%4d)\n", si->ndirty_dent, si->ndirty_dirs, si->ndirty_all); seq_printf(s, " - data: %4d in files:%4d\n", diff --git a/fs/f2fs/dir.c b/fs/f2fs/dir.c index fd0e2cd31a81..20be3a28509b 100644 --- a/fs/f2fs/dir.c +++ b/fs/f2fs/dir.c @@ -282,7 +282,7 @@ struct f2fs_dir_entry *f2fs_find_target_dentry(const struct f2fs_dentry_ptr *d, static struct f2fs_dir_entry *find_in_level(struct inode *dir, unsigned int level, const struct f2fs_filename *fname, - struct folio **res_folio, + void **dentry_block, bool use_hash) { int s = GET_DENTRY_SLOTS(fname->disk_name.len); @@ -313,7 +313,7 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir, bidx = next_pgofs; continue; } else { - *res_folio = dentry_folio; + *dentry_block = dentry_folio; break; } } @@ -321,11 +321,11 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir, de = find_in_block(dir, dentry_folio, fname, &max_slots, use_hash); if (IS_ERR(de)) { f2fs_folio_put(dentry_folio, false); - *res_folio = ERR_CAST(de); + *dentry_block = ERR_CAST(de); de = NULL; break; } else if (de) { - *res_folio = dentry_folio; + *dentry_block = dentry_folio; break; } @@ -352,7 +352,7 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir, struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir, const struct f2fs_filename *fname, - struct folio **res_folio) + void **dentry_block) { unsigned long npages = dir_blocks(dir); struct f2fs_dir_entry *de = NULL; @@ -360,13 +360,13 @@ struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir, unsigned int level; bool use_hash = true; - *res_folio = NULL; + *dentry_block = NULL; #if IS_ENABLED(CONFIG_UNICODE) start_find_entry: #endif if (f2fs_has_inline_dentry(dir)) { - de = f2fs_find_in_inline_dir(dir, fname, res_folio, use_hash); + de = f2fs_find_in_inline_dir(dir, fname, dentry_block, use_hash); goto out; } @@ -382,8 +382,8 @@ struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir, } for (level = 0; level < max_depth; level++) { - de = find_in_level(dir, level, fname, res_folio, use_hash); - if (de || IS_ERR(*res_folio)) + de = find_in_level(dir, level, fname, dentry_block, use_hash); + if (de || IS_ERR(*dentry_block)) break; } @@ -408,7 +408,7 @@ struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir, * Entry is guaranteed to be valid. */ struct f2fs_dir_entry *f2fs_find_entry(struct inode *dir, - const struct qstr *child, struct folio **res_folio) + const struct qstr *child, void **dentry_block) { struct f2fs_dir_entry *de = NULL; struct f2fs_filename fname; @@ -417,67 +417,78 @@ struct f2fs_dir_entry *f2fs_find_entry(struct inode *dir, err = f2fs_setup_filename(dir, child, 1, &fname); if (err) { if (err == -ENOENT) - *res_folio = NULL; + *dentry_block = NULL; else - *res_folio = ERR_PTR(err); + *dentry_block = ERR_PTR(err); return NULL; } - de = __f2fs_find_entry(dir, &fname, res_folio); + de = __f2fs_find_entry(dir, &fname, dentry_block); f2fs_free_filename(&fname); return de; } -struct f2fs_dir_entry *f2fs_parent_dir(struct inode *dir, struct folio **f) +struct f2fs_dir_entry *f2fs_parent_dir(struct inode *dir, void **dentry_block) { - return f2fs_find_entry(dir, &dotdot_name, f); + return f2fs_find_entry(dir, &dotdot_name, dentry_block); } ino_t f2fs_inode_by_name(struct inode *dir, const struct qstr *qstr, - struct folio **folio) + void **dentry_block) { ino_t res = 0; struct f2fs_dir_entry *de; - de = f2fs_find_entry(dir, qstr, folio); + de = f2fs_find_entry(dir, qstr, dentry_block); if (de) { res = le32_to_cpu(de->ino); - f2fs_folio_put(*folio, false); + f2fs_put_dentry_block(*dentry_block, false); } return res; } void f2fs_set_link(struct inode *dir, struct f2fs_dir_entry *de, - struct folio *folio, struct inode *inode) + void *dentry_block, struct inode *inode) { - enum page_type type = f2fs_has_inline_dentry(dir) ? NODE : DATA; + if (f2fs_dentry_is_cache(dentry_block)) { + struct f2fs_cached_block *entry = + f2fs_dentry_cache(dentry_block); + + f2fs_lock_cache(entry); + f2fs_cache_wait_writeback(entry); + de->ino = cpu_to_le32(inode->i_ino); + de->file_type = fs_umode_to_ftype(inode->i_mode); + f2fs_mark_cache_dirty(entry); + } else { + struct folio *folio = f2fs_dentry_folio(dentry_block); - folio_lock(folio); - f2fs_folio_wait_writeback(folio, type, true, true); - de->ino = cpu_to_le32(inode->i_ino); - de->file_type = fs_umode_to_ftype(inode->i_mode); - folio_mark_dirty(folio); + folio_lock(folio); + f2fs_folio_wait_writeback(folio, true, true); + de->ino = cpu_to_le32(inode->i_ino); + de->file_type = fs_umode_to_ftype(inode->i_mode); + folio_mark_dirty(folio); + } inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir)); f2fs_mark_inode_dirty_sync(dir, true); - f2fs_folio_put(folio, true); + f2fs_put_dentry_block(dentry_block, true); } static void init_dent_inode(struct inode *dir, struct inode *inode, const struct f2fs_filename *fname, - struct folio *ifolio) + struct f2fs_cached_block *ientry) { struct f2fs_inode *ri; if (!fname) /* tmpfile case? */ return; - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_cache_wait_writeback(ientry); /* copy name info. to this inode folio */ - ri = F2FS_INODE(ifolio); + ri = F2FS_INODE(ientry); ri->i_namelen = cpu_to_le32(fname->disk_name.len); memcpy(ri->i_name, fname->disk_name.name, fname->disk_name.len); if (IS_ENCRYPTED(dir)) { @@ -498,7 +509,7 @@ static void init_dent_inode(struct inode *dir, struct inode *inode, file_lost_pino(inode); } } - folio_mark_dirty(ifolio); + f2fs_mark_cache_dirty(ientry); } void f2fs_do_make_empty_dir(struct inode *inode, struct inode *parent, @@ -515,16 +526,16 @@ void f2fs_do_make_empty_dir(struct inode *inode, struct inode *parent, } static int make_empty_dir(struct inode *inode, - struct inode *parent, struct folio *folio) + struct inode *parent, struct f2fs_cached_block *ientry) { struct folio *dentry_folio; struct f2fs_dentry_block *dentry_blk; struct f2fs_dentry_ptr d; if (f2fs_has_inline_dentry(inode)) - return f2fs_make_empty_inline_dir(inode, parent, folio); + return f2fs_make_empty_inline_dir(inode, parent, ientry); - dentry_folio = f2fs_get_new_data_folio(inode, folio, 0, true); + dentry_folio = f2fs_get_new_data_folio(inode, ientry, 0, true); if (IS_ERR(dentry_folio)) return PTR_ERR(dentry_folio); @@ -538,50 +549,50 @@ static int make_empty_dir(struct inode *inode, return 0; } -struct folio *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir, - const struct f2fs_filename *fname, struct folio *dfolio) +struct f2fs_cached_block *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir, + const struct f2fs_filename *fname, struct f2fs_cached_block *dentry) { - struct folio *folio; + struct f2fs_cached_block *ientry; int err; if (is_inode_flag_set(inode, FI_NEW_INODE)) { - folio = f2fs_new_inode_folio(inode); - if (IS_ERR(folio)) - return folio; + ientry = f2fs_new_inode_cache(inode); + if (IS_ERR(ientry)) + return ientry; if (S_ISDIR(inode->i_mode)) { /* in order to handle error case */ - folio_get(folio); - err = make_empty_dir(inode, dir, folio); + f2fs_cache_get(ientry); + err = make_empty_dir(inode, dir, ientry); if (err) { - folio_lock(folio); + f2fs_lock_cache(ientry); goto put_error; } - folio_put(folio); + f2fs_put_cache(ientry, false); } - err = f2fs_init_acl(inode, dir, folio, dfolio); + err = f2fs_init_acl(inode, dir, ientry, dentry); if (err) goto put_error; err = f2fs_init_security(inode, dir, fname ? fname->usr_fname : NULL, - folio); + ientry); if (err) goto put_error; if (IS_ENCRYPTED(inode)) { - err = fscrypt_set_context(inode, folio); + err = fscrypt_set_context(inode, ientry); if (err) goto put_error; } } else { - folio = f2fs_get_inode_folio(F2FS_I_SB(dir), inode->i_ino); - if (IS_ERR(folio)) - return folio; + ientry = f2fs_get_inode_cache(F2FS_I_SB(dir), inode->i_ino); + if (IS_ERR(ientry)) + return ientry; } - init_dent_inode(dir, inode, fname, folio); + init_dent_inode(dir, inode, fname, ientry); /* * This file should be checkpointed during fsync. @@ -598,12 +609,12 @@ struct folio *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir, f2fs_remove_orphan_inode(F2FS_I_SB(dir), inode->i_ino); f2fs_i_links_write(inode, true); } - return folio; + return ientry; put_error: clear_nlink(inode); - f2fs_update_inode(inode, folio); - f2fs_folio_put(folio, true); + f2fs_update_inode(inode, ientry); + f2fs_put_cache(ientry, true); return ERR_PTR(err); } @@ -645,14 +656,14 @@ int f2fs_room_for_filename(const void *bitmap, int slots, int max_slots) goto next; } -bool f2fs_has_enough_room(struct inode *dir, struct folio *ifolio, +bool f2fs_has_enough_room(struct inode *dir, struct f2fs_cached_block *ientry, const struct f2fs_filename *fname) { struct f2fs_dentry_ptr d; unsigned int bit_pos; int slots = GET_DENTRY_SLOTS(fname->disk_name.len); - make_dentry_ptr_inline(dir, &d, inline_data_addr(dir, ifolio)); + make_dentry_ptr_inline(dir, &d, inline_data_addr(dir, ientry)); bit_pos = f2fs_room_for_filename(d.bitmap, slots, d.max); @@ -692,7 +703,7 @@ int f2fs_add_regular_entry(struct inode *dir, const struct f2fs_filename *fname, struct folio *dentry_folio = NULL; struct f2fs_dentry_block *dentry_blk = NULL; struct f2fs_dentry_ptr d; - struct folio *folio = NULL; + struct f2fs_cached_block *entry = NULL; int slots, err = 0; level = 0; @@ -739,13 +750,13 @@ int f2fs_add_regular_entry(struct inode *dir, const struct f2fs_filename *fname, ++level; goto start; add_dentry: - f2fs_folio_wait_writeback(dentry_folio, DATA, true, true); + f2fs_folio_wait_writeback(dentry_folio, true, true); if (inode) { f2fs_down_write(&F2FS_I(inode)->i_sem); - folio = f2fs_init_inode_metadata(inode, dir, fname, NULL); - if (IS_ERR(folio)) { - err = PTR_ERR(folio); + entry = f2fs_init_inode_metadata(inode, dir, fname, NULL); + if (IS_ERR(entry)) { + err = PTR_ERR(entry); goto fail; } } @@ -761,9 +772,9 @@ int f2fs_add_regular_entry(struct inode *dir, const struct f2fs_filename *fname, /* synchronize inode page's data from inode cache */ if (is_inode_flag_set(inode, FI_NEW_INODE)) - f2fs_update_inode(inode, folio); + f2fs_update_inode(inode, entry); - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); } f2fs_update_parent_metadata(dir, inode, current_depth); @@ -805,7 +816,7 @@ int f2fs_do_add_link(struct inode *dir, const struct qstr *name, struct inode *inode, nid_t ino, umode_t mode) { struct f2fs_filename fname; - struct folio *folio = NULL; + void *dentry_blk = NULL; struct f2fs_dir_entry *de = NULL; int err; @@ -821,14 +832,14 @@ int f2fs_do_add_link(struct inode *dir, const struct qstr *name, * consistency more. */ if (current != F2FS_I(dir)->task) { - de = __f2fs_find_entry(dir, &fname, &folio); + de = __f2fs_find_entry(dir, &fname, &dentry_blk); F2FS_I(dir)->task = NULL; } if (de) { - f2fs_folio_put(folio, false); + f2fs_put_dentry_block(dentry_blk, false); err = -EEXIST; - } else if (IS_ERR(folio)) { - err = PTR_ERR(folio); + } else if (IS_ERR(dentry_blk)) { + err = PTR_ERR(dentry_blk); } else { err = f2fs_add_dentry(dir, &fname, inode, ino, mode); } @@ -839,16 +850,16 @@ int f2fs_do_add_link(struct inode *dir, const struct qstr *name, int f2fs_do_tmpfile(struct inode *inode, struct inode *dir, struct f2fs_filename *fname) { - struct folio *folio; + struct f2fs_cached_block *ientry; int err = 0; f2fs_down_write(&F2FS_I(inode)->i_sem); - folio = f2fs_init_inode_metadata(inode, dir, fname, NULL); - if (IS_ERR(folio)) { - err = PTR_ERR(folio); + ientry = f2fs_init_inode_metadata(inode, dir, fname, NULL); + if (IS_ERR(ientry)) { + err = PTR_ERR(ientry); goto fail; } - f2fs_folio_put(folio, true); + f2fs_put_cache(ientry, true); clear_inode_flag(inode, FI_NEW_INODE); f2fs_update_time(F2FS_I_SB(inode), REQ_TIME); @@ -884,13 +895,14 @@ void f2fs_drop_nlink(struct inode *dir, struct inode *inode) * It only removes the dentry from the dentry page, corresponding name * entry in name page does not need to be touched during deletion. */ -void f2fs_delete_entry(struct f2fs_dir_entry *dentry, struct folio *folio, +void f2fs_delete_entry(struct f2fs_dir_entry *dentry, void *dentry_block, struct inode *dir, struct inode *inode) { struct f2fs_dentry_block *dentry_blk; + struct folio *folio; unsigned int bit_pos; int slots = GET_DENTRY_SLOTS(le16_to_cpu(dentry->name_len)); - pgoff_t index = folio->index; + pgoff_t index; int i; f2fs_update_time(F2FS_I_SB(dir), REQ_TIME); @@ -899,10 +911,14 @@ void f2fs_delete_entry(struct f2fs_dir_entry *dentry, struct folio *folio, f2fs_add_ino_entry(F2FS_I_SB(dir), dir->i_ino, TRANS_DIR_INO); if (f2fs_has_inline_dentry(dir)) - return f2fs_delete_inline_entry(dentry, folio, dir, inode); + return f2fs_delete_inline_entry(dentry, + f2fs_dentry_cache(dentry_block), dir, inode); + + folio = f2fs_dentry_folio(dentry_block); + index = folio->index; folio_lock(folio); - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); dentry_blk = folio_address(folio); bit_pos = dentry - dentry_blk->dentry; diff --git a/fs/f2fs/extent_cache.c b/fs/f2fs/extent_cache.c index 37cf9fa8d537..e52527c68c02 100644 --- a/fs/f2fs/extent_cache.c +++ b/fs/f2fs/extent_cache.c @@ -20,10 +20,10 @@ #include "segment.h" #include <trace/events/f2fs.h> -bool sanity_check_extent_cache(struct inode *inode, struct folio *ifolio) +bool sanity_check_extent_cache(struct inode *inode, struct f2fs_cached_block *ientry) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - struct f2fs_extent *i_ext = &F2FS_INODE(ifolio)->i_ext; + struct f2fs_extent *i_ext = &F2FS_INODE(ientry)->i_ext; struct extent_info ei; int devi; @@ -416,11 +416,11 @@ static void __drop_largest_extent(struct extent_tree *et, } } -void f2fs_init_read_extent_tree(struct inode *inode, struct folio *ifolio) +void f2fs_init_read_extent_tree(struct inode *inode, struct f2fs_cached_block *ientry) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct extent_tree_info *eti = &sbi->extent_tree[EX_READ]; - struct f2fs_extent *i_ext = &F2FS_INODE(ifolio)->i_ext; + struct f2fs_extent *i_ext = &F2FS_INODE(ientry)->i_ext; struct extent_tree *et; struct extent_node *en; struct extent_info ei = {0}; @@ -428,9 +428,9 @@ void f2fs_init_read_extent_tree(struct inode *inode, struct folio *ifolio) if (!__may_extent_tree(inode, EX_READ)) { /* drop largest read extent */ if (i_ext->len) { - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_cache_wait_writeback(ientry); i_ext->len = 0; - folio_mark_dirty(ifolio); + f2fs_mark_cache_dirty(ientry); } set_inode_flag(inode, FI_NO_EXTENT); return; @@ -956,7 +956,7 @@ static void __update_extent_cache(struct dnode_of_data *dn, enum extent_type typ if (!__may_extent_tree(dn->inode, type)) return; - ei.fofs = f2fs_start_bidx_of_node(ofs_of_node(dn->node_folio), dn->inode) + + ei.fofs = f2fs_start_bidx_of_node(ofs_of_node(dn->node_entry), dn->inode) + dn->ofs_in_node; ei.len = 1; diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 4a811e9d325b..4dd165ac05c9 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -416,7 +416,7 @@ struct inode_entry { struct fsync_node_entry { struct list_head list; /* list head */ - struct folio *folio; /* warm node folio pointer */ + struct f2fs_cached_block *entry; /* warm node cache entry pointer */ unsigned int seq_id; /* sequence id */ }; @@ -1123,11 +1123,11 @@ struct f2fs_nm_info { */ struct dnode_of_data { struct inode *inode; /* vfs inode pointer */ - struct folio *inode_folio; /* its inode folio, NULL is possible */ - struct folio *node_folio; /* cached direct node folio */ + struct f2fs_cached_block *inode_entry; /* generic cache inode entry */ + struct f2fs_cached_block *node_entry; /* generic cache node entry */ nid_t nid; /* node id of the direct node block */ unsigned int ofs_in_node; /* data offset in the node page */ - bool inode_folio_locked; /* inode folio is locked or not */ + bool inode_entry_locked; /* inode entry is locked or not */ bool node_changed; /* is node block changed */ char cur_level; /* level of hole node page */ char max_level; /* level of current page located */ @@ -1135,12 +1135,12 @@ struct dnode_of_data { }; static inline void set_new_dnode(struct dnode_of_data *dn, struct inode *inode, - struct folio *ifolio, struct folio *nfolio, nid_t nid) + struct f2fs_cached_block *ientry, struct f2fs_cached_block *nentry, nid_t nid) { memset(dn, 0, sizeof(*dn)); dn->inode = inode; - dn->inode_folio = ifolio; - dn->node_folio = nfolio; + dn->inode_entry = ientry; + dn->node_entry = nentry; dn->nid = nid; } @@ -1622,10 +1622,9 @@ static inline void f2fs_clear_bit(unsigned int nr, char *addr); * | bit0 = 1 | bit1 | bit2 | ... | bit MAX | private data .... | * bit 0 PAGE_PRIVATE_NOT_POINTER * bit 1 PAGE_PRIVATE_ONGOING_MIGRATION - * bit 2 PAGE_PRIVATE_INLINE_INODE - * bit 3 PAGE_PRIVATE_REF_RESOURCE - * bit 4 PAGE_PRIVATE_ATOMIC_WRITE - * bit 5- f2fs private data + * bit 2 PAGE_PRIVATE_REF_RESOURCE + * bit 3 PAGE_PRIVATE_ATOMIC_WRITE + * bit 4- f2fs private data * * Layout B: lowest bit should be 0 * page.private is a wrapped pointer. @@ -1633,7 +1632,6 @@ static inline void f2fs_clear_bit(unsigned int nr, char *addr); enum { PAGE_PRIVATE_NOT_POINTER, /* private contains non-pointer data */ PAGE_PRIVATE_ONGOING_MIGRATION, /* data page which is on-going migrating */ - PAGE_PRIVATE_INLINE_INODE, /* inode page contains inline data */ PAGE_PRIVATE_REF_RESOURCE, /* dirty page has referenced resources */ PAGE_PRIVATE_ATOMIC_WRITE, /* data page from atomic write path */ PAGE_PRIVATE_MAX @@ -1810,7 +1808,6 @@ struct f2fs_sb_info { /* for node-related operations */ struct f2fs_nm_info *nm_info; /* node manager */ - struct inode *node_inode; /* cache node blocks */ /* for segment-related operations */ struct f2fs_sm_info *sm_info; /* segment manager */ @@ -2283,14 +2280,14 @@ static inline struct f2fs_checkpoint *F2FS_CKPT(struct f2fs_sb_info *sbi) return (struct f2fs_checkpoint *)(sbi->ckpt); } -static inline struct f2fs_node *F2FS_NODE(const struct folio *folio) +static inline struct f2fs_node *F2FS_NODE(const struct f2fs_cached_block *entry) { - return (struct f2fs_node *)folio_address(folio); + return (struct f2fs_node *)CACHED_NODE(entry); } -static inline struct f2fs_inode *F2FS_INODE(const struct folio *folio) +static inline struct f2fs_inode *F2FS_INODE(const struct f2fs_cached_block *entry) { - return &((struct f2fs_node *)folio_address(folio))->i; + return &CACHED_NODE(entry)->i; } static inline struct f2fs_nm_info *NM_I(struct f2fs_sb_info *sbi) @@ -2323,14 +2320,9 @@ static inline bool f2fs_is_meta_cache(struct f2fs_cached_block *entry) return entry->cache && entry->cache == META_CACHE(entry->cache->sbi); } -static inline struct address_space *NODE_MAPPING(struct f2fs_sb_info *sbi) +static inline bool f2fs_is_node_cache(struct f2fs_cached_block *entry) { - return sbi->node_inode->i_mapping; -} - -static inline bool is_node_folio(struct folio *folio) -{ - return folio->mapping == NODE_MAPPING(F2FS_F_SB(folio)); + return entry->cache && entry->cache == NODE_CACHE(entry->cache->sbi); } static inline struct f2fs_bio *F2FS_BIO(struct bio *bio) @@ -2753,17 +2745,14 @@ static inline void clear_page_private_##name(struct page *page) \ } PAGE_PRIVATE_GET_FUNC(nonpointer, NOT_POINTER); -PAGE_PRIVATE_GET_FUNC(inline, INLINE_INODE); PAGE_PRIVATE_GET_FUNC(gcing, ONGOING_MIGRATION); PAGE_PRIVATE_GET_FUNC(atomic, ATOMIC_WRITE); PAGE_PRIVATE_SET_FUNC(reference, REF_RESOURCE); -PAGE_PRIVATE_SET_FUNC(inline, INLINE_INODE); PAGE_PRIVATE_SET_FUNC(gcing, ONGOING_MIGRATION); PAGE_PRIVATE_SET_FUNC(atomic, ATOMIC_WRITE); PAGE_PRIVATE_CLEAR_FUNC(reference, REF_RESOURCE); -PAGE_PRIVATE_CLEAR_FUNC(inline, INLINE_INODE); PAGE_PRIVATE_CLEAR_FUNC(gcing, ONGOING_MIGRATION); PAGE_PRIVATE_CLEAR_FUNC(atomic, ATOMIC_WRITE); @@ -3172,14 +3161,51 @@ static inline void f2fs_put_page(struct page *page, bool unlock) f2fs_folio_put(page_folio(page), unlock); } +#define F2FS_DENTRY_TAG_CACHE 1UL +#define F2FS_DENTRY_TAG_MASK 1UL + +static inline void *f2fs_cache_make_dentry_block(struct f2fs_cached_block *entry) +{ + if (IS_ERR_OR_NULL(entry)) + return entry; + return (void *)((unsigned long)entry | F2FS_DENTRY_TAG_CACHE); +} + +static inline bool f2fs_dentry_is_cache(void *dentry_block) +{ + return ((unsigned long)dentry_block & F2FS_DENTRY_TAG_MASK) == + F2FS_DENTRY_TAG_CACHE; +} + +static inline struct f2fs_cached_block *f2fs_dentry_cache(void *dentry_block) +{ + return (struct f2fs_cached_block *) + ((unsigned long)dentry_block & ~F2FS_DENTRY_TAG_MASK); +} + +static inline struct folio *f2fs_dentry_folio(void *dentry_block) +{ + return (struct folio *)dentry_block; +} + +static inline void f2fs_put_dentry_block(void *dentry_block, bool unlock) +{ + if (IS_ERR_OR_NULL(dentry_block)) + return; + if (f2fs_dentry_is_cache(dentry_block)) + f2fs_put_cache(f2fs_dentry_cache(dentry_block), unlock); + else + f2fs_folio_put(f2fs_dentry_folio(dentry_block), unlock); +} + static inline void f2fs_put_dnode(struct dnode_of_data *dn) { - if (dn->node_folio) - f2fs_folio_put(dn->node_folio, true); - if (dn->inode_folio && dn->node_folio != dn->inode_folio) - f2fs_folio_put(dn->inode_folio, false); - dn->node_folio = NULL; - dn->inode_folio = NULL; + if (dn->node_entry) + f2fs_put_cache(dn->node_entry, true); + if (dn->inode_entry && dn->node_entry != dn->inode_entry) + f2fs_put_cache(dn->inode_entry, false); + dn->node_entry = NULL; + dn->inode_entry = NULL; } static inline struct kmem_cache *f2fs_kmem_cache_create(const char *name, @@ -3270,9 +3296,9 @@ static inline void f2fs_radix_tree_insert(struct radix_tree_root *root, #define RAW_IS_INODE(p) ((p)->footer.nid == (p)->footer.ino) -static inline bool IS_INODE(const struct folio *folio) +static inline bool IS_INODE(const struct f2fs_cached_block *entry) { - struct f2fs_node *p = F2FS_NODE(folio); + struct f2fs_node *p = F2FS_NODE(entry); return RAW_IS_INODE(p); } @@ -3289,32 +3315,34 @@ static inline __le32 *blkaddr_in_node(struct f2fs_node *node) } static inline int f2fs_has_extra_attr(struct inode *inode); + static inline unsigned int get_dnode_base(struct inode *inode, - struct folio *node_folio) + const struct f2fs_cached_block *entry) { - if (!IS_INODE(node_folio)) + if (!IS_INODE(entry)) return 0; return inode ? get_extra_isize(inode) : - offset_in_addr(&F2FS_NODE(node_folio)->i); + offset_in_addr(&CACHED_NODE(entry)->i); } static inline __le32 *get_dnode_addr(struct inode *inode, - struct folio *node_folio) + const struct f2fs_cached_block *entry) { - return blkaddr_in_node(F2FS_NODE(node_folio)) + - get_dnode_base(inode, node_folio); + return blkaddr_in_node(CACHED_NODE(entry)) + + get_dnode_base(inode, entry); } static inline block_t data_blkaddr(struct inode *inode, - struct folio *node_folio, unsigned int offset) + const struct f2fs_cached_block *entry, + unsigned int offset) { - return le32_to_cpu(*(get_dnode_addr(inode, node_folio) + offset)); + return le32_to_cpu(*(get_dnode_addr(inode, entry) + offset)); } static inline block_t f2fs_data_blkaddr(struct dnode_of_data *dn) { - return data_blkaddr(dn->inode, dn->node_folio, dn->ofs_in_node); + return data_blkaddr(dn->inode, dn->node_entry, dn->ofs_in_node); } static inline int f2fs_test_bit(unsigned int nr, char *addr) @@ -3625,10 +3653,10 @@ static inline unsigned int addrs_per_page(struct inode *inode, return addrs; } -static inline -void *inline_xattr_addr(struct inode *inode, const struct folio *folio) +static inline void *inline_xattr_addr(struct inode *inode, + const struct f2fs_cached_block *entry) { - struct f2fs_inode *ri = F2FS_INODE(folio); + struct f2fs_inode *ri = F2FS_INODE(entry); return (void *)&(ri->i_addr[DEF_ADDRS_PER_INODE - get_inline_xattr_addrs(inode)]); @@ -3675,9 +3703,10 @@ static inline bool f2fs_is_cow_file(struct inode *inode) return is_inode_flag_set(inode, FI_COW_FILE); } -static inline void *inline_data_addr(struct inode *inode, struct folio *folio) +static inline void *inline_data_addr(struct inode *inode, + const struct f2fs_cached_block *entry) { - __le32 *addr = get_dnode_addr(inode, folio); + __le32 *addr = get_dnode_addr(inode, entry); return (void *)(addr + DEF_INLINE_RESERVED_SIZE); } @@ -3887,12 +3916,12 @@ int f2fs_pin_file_control(struct inode *inode, bool inc); * inode.c */ void f2fs_set_inode_flags(struct inode *inode); -bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct folio *folio); -void f2fs_inode_chksum_set(struct f2fs_sb_info *sbi, struct folio *folio); +bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry); +void f2fs_inode_chksum_set(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry); struct inode *f2fs_iget(struct super_block *sb, unsigned long ino); struct inode *f2fs_iget_retry(struct super_block *sb, unsigned long ino); int f2fs_try_to_free_nats(struct f2fs_sb_info *sbi, int nr_shrink); -void f2fs_update_inode(struct inode *inode, struct folio *node_folio); +void f2fs_update_inode(struct inode *inode, struct f2fs_cached_block *entry); void f2fs_update_inode_page(struct inode *inode); int f2fs_write_inode(struct inode *inode, struct writeback_control *wbc); void f2fs_remove_donate_inode(struct inode *inode); @@ -3941,22 +3970,22 @@ int f2fs_fill_dentries(struct dir_context *ctx, struct f2fs_dentry_ptr *d, unsigned int start_pos, struct fscrypt_str *fstr); void f2fs_do_make_empty_dir(struct inode *inode, struct inode *parent, struct f2fs_dentry_ptr *d); -struct folio *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir, - const struct f2fs_filename *fname, struct folio *dfolio); +struct f2fs_cached_block *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir, + const struct f2fs_filename *fname, struct f2fs_cached_block *dentry); void f2fs_update_parent_metadata(struct inode *dir, struct inode *inode, unsigned int current_depth); int f2fs_room_for_filename(const void *bitmap, int slots, int max_slots); void f2fs_drop_nlink(struct inode *dir, struct inode *inode); struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir, - const struct f2fs_filename *fname, struct folio **res_folio); + const struct f2fs_filename *fname, void **dentry_block); struct f2fs_dir_entry *f2fs_find_entry(struct inode *dir, - const struct qstr *child, struct folio **res_folio); -struct f2fs_dir_entry *f2fs_parent_dir(struct inode *dir, struct folio **f); + const struct qstr *child, void **dentry_block); +struct f2fs_dir_entry *f2fs_parent_dir(struct inode *dir, void **dentry_block); ino_t f2fs_inode_by_name(struct inode *dir, const struct qstr *qstr, - struct folio **folio); + void **dentry_block); void f2fs_set_link(struct inode *dir, struct f2fs_dir_entry *de, - struct folio *folio, struct inode *inode); -bool f2fs_has_enough_room(struct inode *dir, struct folio *ifolio, + void *dentry_blk, struct inode *inode); +bool f2fs_has_enough_room(struct inode *dir, struct f2fs_cached_block *ientry, const struct f2fs_filename *fname); void f2fs_update_dentry(nid_t ino, umode_t mode, struct f2fs_dentry_ptr *d, const struct fscrypt_str *name, f2fs_hash_t name_hash, @@ -3967,7 +3996,7 @@ int f2fs_add_dentry(struct inode *dir, const struct f2fs_filename *fname, struct inode *inode, nid_t ino, umode_t mode); int f2fs_do_add_link(struct inode *dir, const struct qstr *name, struct inode *inode, nid_t ino, umode_t mode); -void f2fs_delete_entry(struct f2fs_dir_entry *dentry, struct folio *folio, +void f2fs_delete_entry(struct f2fs_dir_entry *dentry, void *dentry_blk, struct inode *dir, struct inode *inode); int f2fs_do_tmpfile(struct inode *inode, struct inode *dir, struct f2fs_filename *fname); @@ -4010,9 +4039,9 @@ enum node_type; int f2fs_check_nid_range(struct f2fs_sb_info *sbi, nid_t nid); bool f2fs_available_free_memory(struct f2fs_sb_info *sbi, int type); -bool f2fs_in_warm_node_list(struct folio *folio); +bool f2fs_in_warm_node_list(struct f2fs_cached_block *entry); void f2fs_init_fsync_node_info(struct f2fs_sb_info *sbi); -void f2fs_del_fsync_node_entry(struct f2fs_sb_info *sbi, struct folio *folio); +void f2fs_del_fsync_node_entry(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry); void f2fs_reset_fsync_node_info(struct f2fs_sb_info *sbi); bool f2fs_need_dentry_mark(struct f2fs_sb_info *sbi, nid_t nid); bool f2fs_is_checkpointed_node(struct f2fs_sb_info *sbi, nid_t nid); @@ -4023,29 +4052,28 @@ pgoff_t f2fs_get_next_page_offset(struct dnode_of_data *dn, pgoff_t pgofs); int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode); int f2fs_truncate_inode_blocks(struct inode *inode, pgoff_t from); int f2fs_truncate_xattr_node(struct inode *inode); -int f2fs_wait_on_node_pages_writeback(struct f2fs_sb_info *sbi, +int f2fs_wait_on_node_caches_writeback(struct f2fs_sb_info *sbi, unsigned int seq_id); +int f2fs_write_node_caches(struct f2fs_sb_info *sbi); int f2fs_remove_inode_page(struct inode *inode); -struct folio *f2fs_new_inode_folio(struct inode *inode); -struct folio *f2fs_new_node_folio(struct dnode_of_data *dn, unsigned int ofs); +struct f2fs_cached_block *f2fs_new_inode_cache(struct inode *inode); +struct f2fs_cached_block *f2fs_new_node_cache(struct dnode_of_data *dn, unsigned int ofs); void f2fs_ra_node_page(struct f2fs_sb_info *sbi, nid_t nid); -struct folio *f2fs_get_node_folio(struct f2fs_sb_info *sbi, pgoff_t nid, +struct f2fs_cached_block *f2fs_get_node_cache(struct f2fs_sb_info *sbi, pgoff_t nid, enum node_type node_type); int f2fs_sanity_check_node_footer(struct f2fs_sb_info *sbi, - struct folio *folio, pgoff_t nid, + struct f2fs_cached_block *entry, pgoff_t nid, enum node_type ntype, bool in_irq); -struct folio *f2fs_get_inode_folio(struct f2fs_sb_info *sbi, pgoff_t ino); -struct folio *f2fs_get_xnode_folio(struct f2fs_sb_info *sbi, pgoff_t xnid); -int f2fs_write_single_node_folio(struct folio *node_folio, int sync_mode, +struct f2fs_cached_block *f2fs_get_inode_cache(struct f2fs_sb_info *sbi, pgoff_t ino); +struct f2fs_cached_block *f2fs_get_xnode_cache(struct f2fs_sb_info *sbi, pgoff_t xnid); +int f2fs_write_node_cache(struct f2fs_cached_block *entry, int sync_mode, bool mark_dirty, enum iostat_type io_type); -int f2fs_move_node_folio(struct folio *node_folio, int gc_type); +int f2fs_move_node_cache(struct f2fs_cached_block *entry, int gc_type); void f2fs_flush_inline_data(struct f2fs_sb_info *sbi); -int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode, - struct writeback_control *wbc, bool atomic, - unsigned int *seq_id); -int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, - struct writeback_control *wbc, - bool do_balance, enum iostat_type io_type); +int f2fs_fsync_node_caches(struct f2fs_sb_info *sbi, struct inode *inode, + bool atomic, unsigned int *seq_id); +int f2fs_writeback_node_caches(struct f2fs_sb_info *sbi, long nr_to_write, + bool sync, bool do_balance, enum iostat_type io_type); int f2fs_build_free_nids(struct f2fs_sb_info *sbi, bool sync, bool mount); bool f2fs_alloc_nid(struct f2fs_sb_info *sbi, nid_t *nid); void f2fs_alloc_nid_done(struct f2fs_sb_info *sbi, nid_t nid); @@ -4124,14 +4152,13 @@ void f2fs_replace_block(struct f2fs_sb_info *sbi, struct dnode_of_data *dn, bool recover_newaddr); enum temp_type f2fs_get_segment_temp(struct f2fs_sb_info *sbi, enum log_type seg_type); -int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, struct folio *folio, +int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, block_t old_blkaddr, block_t *new_blkaddr, struct f2fs_summary *sum, int type, struct f2fs_io_info *fio); void f2fs_update_device_state(struct f2fs_sb_info *sbi, nid_t ino, block_t blkaddr, unsigned int blkcnt); -void f2fs_folio_wait_writeback(struct folio *folio, enum page_type type, - bool ordered, bool locked); +void f2fs_folio_wait_writeback(struct folio *folio, bool ordered, bool locked); void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr); void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr, block_t len); @@ -4252,14 +4279,14 @@ void f2fs_destroy_bio_entry_cache(void); void f2fs_submit_read_bio(struct f2fs_sb_info *sbi, struct bio *bio, enum page_type type); int f2fs_init_write_merge_io(struct f2fs_sb_info *sbi); -void f2fs_submit_merged_write(struct f2fs_sb_info *sbi, enum page_type type); void f2fs_submit_merged_write_cond(struct f2fs_sb_info *sbi, - struct inode *inode, struct folio *folio, - nid_t ino, enum page_type type); + struct inode *inode, struct folio *folio); void f2fs_submit_merged_write_folio(struct f2fs_sb_info *sbi, - struct folio *folio, enum page_type type); -bool f2fs_submit_merged_write_cache(struct f2fs_cached_block *entry, - enum page_type type); + struct folio *folio); +bool f2fs_submit_merged_write_cache(struct f2fs_sb_info *sbi, + struct f2fs_cached_block *entry, + nid_t ino, enum page_type type); +void f2fs_submit_merged_write(struct f2fs_sb_info *sbi, enum page_type type); void f2fs_submit_merged_ipu_write(struct f2fs_sb_info *sbi, struct bio **bio, struct folio *folio); void f2fs_submit_all_merged_ipu_writes(struct f2fs_sb_info *sbi); @@ -4285,7 +4312,7 @@ struct folio *f2fs_find_data_folio(struct inode *inode, pgoff_t index, struct folio *f2fs_get_lock_data_folio(struct inode *inode, pgoff_t index, bool for_write); struct folio *f2fs_get_new_data_folio(struct inode *inode, - struct folio *ifolio, pgoff_t index, bool new_i_size); + struct f2fs_cached_block *ientry, pgoff_t index, bool new_i_size); int f2fs_do_write_data_page(struct f2fs_io_info *fio); int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map, int flag); int f2fs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, @@ -4398,7 +4425,7 @@ struct f2fs_stat_info { unsigned int bimodal, avg_vblocks; int util_free, util_valid, util_invalid; int rsvd_segs, overp_segs; - int dirty_count, node_pages, meta_caches, compress_pages; + int dirty_count, node_caches, meta_caches, compress_pages; int compress_page_hit; int prefree_count, free_segs, free_secs; int cp_call_count[MAX_CALL_TYPE], cp_count; @@ -4599,7 +4626,6 @@ extern const struct file_operations f2fs_dir_operations; extern const struct file_operations f2fs_file_operations; extern const struct inode_operations f2fs_file_inode_operations; extern const struct address_space_operations f2fs_dblock_aops; -extern const struct address_space_operations f2fs_node_aops; extern const struct inode_operations f2fs_dir_inode_operations; extern const struct inode_operations f2fs_symlink_inode_operations; extern const struct inode_operations f2fs_encrypted_symlink_inode_operations; @@ -4610,10 +4636,10 @@ extern struct kmem_cache *f2fs_inode_entry_slab; * inline.c */ bool f2fs_may_inline_data(struct inode *inode); -bool f2fs_sanity_check_inline_data(struct inode *inode, struct folio *ifolio); +bool f2fs_sanity_check_inline_data(struct inode *inode, struct f2fs_cached_block *ientry); bool f2fs_may_inline_dentry(struct inode *inode); -void f2fs_do_read_inline_data(struct folio *folio, struct folio *ifolio); -void f2fs_truncate_inline_inode(struct inode *inode, struct folio *ifolio, +void f2fs_do_read_inline_data(struct folio *folio, struct f2fs_cached_block *ientry); +void f2fs_truncate_inline_inode(struct inode *inode, struct f2fs_cached_block *ientry, u64 from); int f2fs_read_inline_data(struct inode *inode, struct folio *folio); int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio); @@ -4622,14 +4648,15 @@ int f2fs_try_convert_inline_dir(struct inode *dir, struct dentry *dentry); int f2fs_write_inline_data(struct inode *inode, struct folio *folio); int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entry); struct f2fs_dir_entry *f2fs_find_in_inline_dir(struct inode *dir, - const struct f2fs_filename *fname, struct folio **res_folio, + const struct f2fs_filename *fname, void **dentry_block, bool use_hash); int f2fs_make_empty_inline_dir(struct inode *inode, struct inode *parent, - struct folio *ifolio); + struct f2fs_cached_block *ientry); int f2fs_add_inline_entry(struct inode *dir, const struct f2fs_filename *fname, struct inode *inode, nid_t ino, umode_t mode); void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry, - struct folio *folio, struct inode *dir, struct inode *inode); + struct f2fs_cached_block *ientry, struct inode *dir, + struct inode *inode); bool f2fs_empty_inline_dir(struct inode *dir); int f2fs_read_inline_dir(struct file *file, struct dir_context *ctx, struct fscrypt_str *fstr); @@ -4652,7 +4679,7 @@ void f2fs_leave_shrinker(struct f2fs_sb_info *sbi); /* * extent_cache.c */ -bool sanity_check_extent_cache(struct inode *inode, struct folio *ifolio); +bool sanity_check_extent_cache(struct inode *inode, struct f2fs_cached_block *ientry); void f2fs_init_extent_tree(struct inode *inode); void f2fs_drop_extent_tree(struct inode *inode); void f2fs_destroy_extent_node(struct inode *inode); @@ -4662,7 +4689,7 @@ int __init f2fs_create_extent_cache(void); void f2fs_destroy_extent_cache(void); /* read extent cache ops */ -void f2fs_init_read_extent_tree(struct inode *inode, struct folio *ifolio); +void f2fs_init_read_extent_tree(struct inode *inode, struct f2fs_cached_block *ientry); bool f2fs_lookup_read_extent_cache(struct inode *inode, pgoff_t pgofs, struct extent_info *ei); bool f2fs_lookup_read_extent_cache_block(struct inode *inode, pgoff_t index, diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c index 92daa41dd96d..71d124f07abd 100644 --- a/fs/f2fs/file.c +++ b/fs/f2fs/file.c @@ -212,7 +212,7 @@ static vm_fault_t f2fs_vm_page_mkwrite(struct vm_fault *vmf) goto out_sem; } - f2fs_folio_wait_writeback(folio, DATA, false, true); + f2fs_folio_wait_writeback(folio, false, true); /* wait for GCed page writeback via generic cache */ f2fs_wait_on_block_writeback(inode, dn.data_blkaddr); @@ -307,13 +307,13 @@ static inline enum cp_reason_type need_do_checkpoint(struct inode *inode) static bool need_inode_page_update(struct f2fs_sb_info *sbi, nid_t ino) { - struct folio *i = filemap_get_folio(NODE_MAPPING(sbi), ino); + struct f2fs_cached_block *entry = f2fs_find_node_cache(sbi, ino); bool ret = false; /* But we need to avoid that there are some inode updates */ - if ((!IS_ERR(i) && folio_test_dirty(i)) || + if ((!IS_ERR(entry) && f2fs_cache_test_dirty(entry)) || f2fs_need_inode_block_update(sbi, ino)) ret = true; - f2fs_folio_put(i, false); + f2fs_put_cache(entry, false); return ret; } @@ -339,10 +339,6 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end, nid_t ino = inode->i_ino; int ret = 0; enum cp_reason_type cp_reason = 0; - struct writeback_control wbc = { - .sync_mode = WB_SYNC_ALL, - .nr_to_write = LONG_MAX, - }; unsigned int seq_id = 0; if (unlikely(f2fs_readonly(inode->i_sb))) @@ -421,7 +417,7 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end, } sync_nodes: atomic_inc(&sbi->wb_sync_req[NODE]); - ret = f2fs_fsync_node_pages(sbi, inode, &wbc, atomic, &seq_id); + ret = f2fs_fsync_node_caches(sbi, inode, atomic, &seq_id); atomic_dec(&sbi->wb_sync_req[NODE]); if (ret) goto out; @@ -447,7 +443,7 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end, * given fsync mark. */ if (!atomic) { - ret = f2fs_wait_on_node_pages_writeback(sbi, seq_id); + ret = f2fs_wait_on_node_caches_writeback(sbi, seq_id); if (ret) goto out; } @@ -484,7 +480,7 @@ static bool __found_offset(struct address_space *mapping, bool compressed_cluster = false; if (f2fs_compressed_file(inode)) { - block_t first_blkaddr = data_blkaddr(dn->inode, dn->node_folio, + block_t first_blkaddr = data_blkaddr(dn->inode, dn->node_entry, ALIGN_DOWN(dn->ofs_in_node, F2FS_I(inode)->i_cluster_size)); compressed_cluster = first_blkaddr == COMPRESS_ADDR; @@ -554,7 +550,7 @@ static loff_t f2fs_seek_block(struct file *file, loff_t offset, int whence) } } - end_offset = ADDRS_PER_PAGE(dn.node_folio, inode); + end_offset = ADDRS_PER_PAGE(dn.node_entry, inode); /* find data/hole in dnode block */ for (; dn.ofs_in_node < end_offset; @@ -716,7 +712,7 @@ void f2fs_truncate_data_blocks_range(struct dnode_of_data *dn, int count) block_t blkstart; int blklen = 0; - addr = get_dnode_addr(dn->inode, dn->node_folio) + ofs; + addr = get_dnode_addr(dn->inode, dn->node_entry) + ofs; blkstart = le32_to_cpu(*addr); /* Assumption: truncation starts with cluster */ @@ -780,7 +776,7 @@ void f2fs_truncate_data_blocks_range(struct dnode_of_data *dn, int count) * once we invalidate valid blkaddr in range [ofs, ofs + count], * we will invalidate all blkaddr in the whole range. */ - fofs = f2fs_start_bidx_of_node(ofs_of_node(dn->node_folio), + fofs = f2fs_start_bidx_of_node(ofs_of_node(dn->node_entry), dn->inode) + ofs; f2fs_update_read_extent_cache_range(dn, fofs, 0, len); f2fs_update_age_extent_cache_range(dn, fofs, len); @@ -818,7 +814,7 @@ static int truncate_partial_data_page(struct inode *inode, u64 from, if (IS_ERR(folio)) return PTR_ERR(folio) == -ENOENT ? 0 : PTR_ERR(folio); truncate_out: - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); folio_zero_segment(folio, offset, folio_size(folio)); /* An encrypted inode should have a key and truncate the last page. */ @@ -836,7 +832,7 @@ int f2fs_do_truncate_blocks(struct inode *inode, u64 from, bool lock) struct f2fs_lock_context lc; pgoff_t free_from; int count = 0, err = 0; - struct folio *ifolio; + struct f2fs_cached_block *ientry; bool truncate_page = false; trace_f2fs_truncate_blocks_enter(inode, from); @@ -854,9 +850,9 @@ int f2fs_do_truncate_blocks(struct inode *inode, u64 from, bool lock) if (lock) f2fs_lock_op(sbi, &lc); - ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(ifolio)) { - err = PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(ientry)) { + err = PTR_ERR(ientry); goto out; } @@ -875,18 +871,18 @@ int f2fs_do_truncate_blocks(struct inode *inode, u64 from, bool lock) f2fs_drop_extent_tree(inode); - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); goto out; } if (f2fs_has_inline_data(inode)) { - f2fs_truncate_inline_inode(inode, ifolio, from); - f2fs_folio_put(ifolio, true); + f2fs_truncate_inline_inode(inode, ientry, from); + f2fs_put_cache(ientry, true); truncate_page = true; goto out; } - set_new_dnode(&dn, inode, ifolio, NULL, 0); + set_new_dnode(&dn, inode, ientry, NULL, 0); err = f2fs_get_dnode_of_data(&dn, free_from, LOOKUP_NODE_RA); if (err) { if (err == -ENOENT) @@ -894,12 +890,12 @@ int f2fs_do_truncate_blocks(struct inode *inode, u64 from, bool lock) goto out; } - count = ADDRS_PER_PAGE(dn.node_folio, inode); + count = ADDRS_PER_PAGE(dn.node_entry, inode); count -= dn.ofs_in_node; f2fs_bug_on(sbi, count < 0); - if (dn.ofs_in_node || IS_INODE(dn.node_folio)) { + if (dn.ofs_in_node || IS_INODE(dn.node_entry)) { f2fs_truncate_data_blocks_range(&dn, count); free_from += count; } @@ -1308,7 +1304,7 @@ static int fill_zero(struct inode *inode, pgoff_t index, if (IS_ERR(folio)) return PTR_ERR(folio); - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); folio_zero_range(folio, start, len); folio_mark_dirty(folio); f2fs_folio_put(folio, true); @@ -1334,7 +1330,7 @@ int f2fs_truncate_hole(struct inode *inode, pgoff_t pg_start, pgoff_t pg_end) return err; } - end_offset = ADDRS_PER_PAGE(dn.node_folio, inode); + end_offset = ADDRS_PER_PAGE(dn.node_entry, inode); count = min(end_offset - dn.ofs_in_node, pg_end - pg_start); f2fs_bug_on(F2FS_I_SB(inode), count == 0 || count > end_offset); @@ -1434,7 +1430,7 @@ static int __read_out_blkaddrs(struct inode *inode, block_t *blkaddr, goto next; } - done = min((pgoff_t)ADDRS_PER_PAGE(dn.node_folio, inode) - + done = min((pgoff_t)ADDRS_PER_PAGE(dn.node_entry, inode) - dn.ofs_in_node, len); for (i = 0; i < done; i++, blkaddr++, do_replace++, dn.ofs_in_node++) { *blkaddr = f2fs_data_blkaddr(&dn); @@ -1523,7 +1519,7 @@ static int __clone_blkaddrs(struct inode *src_inode, struct inode *dst_inode, } ilen = min((pgoff_t) - ADDRS_PER_PAGE(dn.node_folio, dst_inode) - + ADDRS_PER_PAGE(dn.node_entry, dst_inode) - dn.ofs_in_node, len - i); do { dn.data_blkaddr = f2fs_data_blkaddr(&dn); @@ -1561,7 +1557,7 @@ static int __clone_blkaddrs(struct inode *src_inode, struct inode *dst_inode, return PTR_ERR(fdst); } - f2fs_folio_wait_writeback(fdst, DATA, true, true); + f2fs_folio_wait_writeback(fdst, true, true); memcpy_folio(fdst, 0, fsrc, 0, PAGE_SIZE); folio_mark_dirty(fdst); @@ -1830,7 +1826,7 @@ static int f2fs_zero_range(struct inode *inode, loff_t offset, loff_t len, goto out; } - end_offset = ADDRS_PER_PAGE(dn.node_folio, inode); + end_offset = ADDRS_PER_PAGE(dn.node_entry, inode); end = min(pg_end, end_offset - dn.ofs_in_node + index); ret = f2fs_do_zero_range(&dn, index, end); @@ -3128,7 +3124,7 @@ static int f2fs_defragment_range(struct f2fs_sb_info *sbi, goto clear_out; } - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); folio_mark_dirty(folio); folio_set_f2fs_gcing(folio); @@ -4162,7 +4158,7 @@ static int release_compress_blocks(struct dnode_of_data *dn, pgoff_t count) int i; for (i = 0; i < count; i++) { - blkaddr = data_blkaddr(dn->inode, dn->node_folio, + blkaddr = data_blkaddr(dn->inode, dn->node_entry, dn->ofs_in_node + i); if (!__is_valid_data_blkaddr(blkaddr)) @@ -4281,7 +4277,7 @@ static int f2fs_release_compress_blocks(struct file *filp, unsigned long arg) break; } - end_offset = ADDRS_PER_PAGE(dn.node_folio, inode); + end_offset = ADDRS_PER_PAGE(dn.node_entry, inode); count = min(end_offset - dn.ofs_in_node, last_idx - page_idx); count = round_up(count, fi->i_cluster_size); @@ -4332,7 +4328,7 @@ static int reserve_compress_blocks(struct dnode_of_data *dn, pgoff_t count, int i; for (i = 0; i < count; i++) { - blkaddr = data_blkaddr(dn->inode, dn->node_folio, + blkaddr = data_blkaddr(dn->inode, dn->node_entry, dn->ofs_in_node + i); if (!__is_valid_data_blkaddr(blkaddr)) @@ -4349,7 +4345,7 @@ static int reserve_compress_blocks(struct dnode_of_data *dn, pgoff_t count, int ret; for (i = 0; i < cluster_size; i++) { - blkaddr = data_blkaddr(dn->inode, dn->node_folio, + blkaddr = data_blkaddr(dn->inode, dn->node_entry, dn->ofs_in_node + i); if (i == 0) { @@ -4460,7 +4456,7 @@ static int f2fs_reserve_compress_blocks(struct file *filp, unsigned long arg) break; } - end_offset = ADDRS_PER_PAGE(dn.node_folio, inode); + end_offset = ADDRS_PER_PAGE(dn.node_entry, inode); count = min(end_offset - dn.ofs_in_node, last_idx - page_idx); count = round_up(count, fi->i_cluster_size); @@ -4626,7 +4622,7 @@ static int f2fs_sec_trim_file(struct file *filp, unsigned long arg) goto out; } - end_offset = ADDRS_PER_PAGE(dn.node_folio, inode); + end_offset = ADDRS_PER_PAGE(dn.node_entry, inode); count = min(end_offset - dn.ofs_in_node, pg_end - index); for (i = 0; i < count; i++, index++, dn.ofs_in_node++) { struct block_device *cur_bdev; @@ -4822,7 +4818,7 @@ static int redirty_blocks(struct inode *inode, pgoff_t page_idx, int len) /* It will never fail, when folio has pinned above */ f2fs_bug_on(F2FS_I_SB(inode), IS_ERR(folio)); - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); folio_mark_dirty(folio); folio_set_f2fs_gcing(folio); diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c index 54327cb2e27e..12787d0414ec 100644 --- a/fs/f2fs/gc.c +++ b/fs/f2fs/gc.c @@ -1054,7 +1054,7 @@ static int gc_node_segment(struct f2fs_sb_info *sbi, for (off = 0; off < usable_blks_in_seg; off++, entry++) { nid_t nid = le32_to_cpu(entry->nid); - struct folio *node_folio; + struct f2fs_cached_block *node_entry; struct node_info ni; int err; @@ -1077,27 +1077,27 @@ static int gc_node_segment(struct f2fs_sb_info *sbi, } /* phase == 2 */ - node_folio = f2fs_get_node_folio(sbi, nid, NODE_TYPE_REGULAR); - if (IS_ERR(node_folio)) + node_entry = f2fs_get_node_cache(sbi, nid, NODE_TYPE_REGULAR); + if (IS_ERR(node_entry)) continue; /* block may become invalid during f2fs_get_node_folio */ if (check_valid_map(sbi, segno, off) == 0) { - f2fs_folio_put(node_folio, true); + f2fs_put_cache(node_entry, true); continue; } if (f2fs_get_node_info(sbi, nid, &ni, false)) { - f2fs_folio_put(node_folio, true); + f2fs_put_cache(node_entry, true); continue; } if (ni.blk_addr != start_addr + off) { - f2fs_folio_put(node_folio, true); + f2fs_put_cache(node_entry, true); continue; } - err = f2fs_move_node_folio(node_folio, gc_type); + err = f2fs_move_node_cache(node_entry, gc_type); if (!err && gc_type == FG_GC) submitted++; stat_inc_node_blk_count(sbi, 1, gc_type); @@ -1146,7 +1146,7 @@ block_t f2fs_start_bidx_of_node(unsigned int node_ofs, struct inode *inode) static bool is_alive(struct f2fs_sb_info *sbi, struct f2fs_summary *sum, struct node_info *dni, block_t blkaddr, unsigned int *nofs) { - struct folio *node_folio; + struct f2fs_cached_block *node_entry; nid_t nid; unsigned int ofs_in_node, max_addrs, base; block_t source_blkaddr; @@ -1154,12 +1154,12 @@ static bool is_alive(struct f2fs_sb_info *sbi, struct f2fs_summary *sum, nid = le32_to_cpu(sum->nid); ofs_in_node = le16_to_cpu(sum->ofs_in_node); - node_folio = f2fs_get_node_folio(sbi, nid, NODE_TYPE_REGULAR); - if (IS_ERR(node_folio)) + node_entry = f2fs_get_node_cache(sbi, nid, NODE_TYPE_REGULAR); + if (IS_ERR(node_entry)) return false; if (f2fs_get_node_info(sbi, nid, dni, false)) { - f2fs_folio_put(node_folio, true); + f2fs_put_cache(node_entry, true); return false; } @@ -1170,12 +1170,12 @@ static bool is_alive(struct f2fs_sb_info *sbi, struct f2fs_summary *sum, } if (f2fs_check_nid_range(sbi, dni->ino)) { - f2fs_folio_put(node_folio, true); + f2fs_put_cache(node_entry, true); return false; } - if (IS_INODE(node_folio)) { - base = offset_in_addr(F2FS_INODE(node_folio)); + if (IS_INODE(node_entry)) { + base = offset_in_addr(F2FS_INODE(node_entry)); max_addrs = DEF_ADDRS_PER_INODE; } else { base = 0; @@ -1185,13 +1185,13 @@ static bool is_alive(struct f2fs_sb_info *sbi, struct f2fs_summary *sum, if (base + ofs_in_node >= max_addrs) { f2fs_err(sbi, "Inconsistent blkaddr offset: base:%u, ofs_in_node:%u, max:%u, ino:%u, nid:%u", base, ofs_in_node, max_addrs, dni->ino, dni->nid); - f2fs_folio_put(node_folio, true); + f2fs_put_cache(node_entry, true); return false; } - *nofs = ofs_of_node(node_folio); - source_blkaddr = data_blkaddr(NULL, node_folio, ofs_in_node); - f2fs_folio_put(node_folio, true); + *nofs = ofs_of_node(node_entry); + source_blkaddr = data_blkaddr(NULL, node_entry, ofs_in_node); + f2fs_put_cache(node_entry, true); if (source_blkaddr != blkaddr) { #ifdef CONFIG_F2FS_CHECK_FS @@ -1283,7 +1283,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index) * don't cache encrypted data into meta inode until previous dirty * data were writebacked to avoid racing between GC and flush. */ - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); f2fs_wait_on_block_writeback(inode, dn.data_blkaddr); @@ -1394,7 +1394,7 @@ static int move_data_block(struct inode *inode, block_t bidx, * don't cache encrypted data into meta inode until previous dirty * data were writebacked to avoid racing between GC and flush. */ - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); f2fs_wait_on_block_writeback(inode, dn.data_blkaddr); @@ -1443,7 +1443,7 @@ static int move_data_block(struct inode *inode, block_t bidx, set_summary(&sum, dn.nid, dn.ofs_in_node, ni.version); /* allocate block address */ - err = f2fs_allocate_data_block(fio.sbi, NULL, fio.old_blkaddr, &newaddr, + err = f2fs_allocate_data_block(fio.sbi, fio.old_blkaddr, &newaddr, &sum, type, NULL); if (err) { f2fs_put_cache(sentry, true); @@ -1545,7 +1545,7 @@ static int move_data_page(struct inode *inode, block_t bidx, int gc_type, bool is_dirty = folio_test_dirty(folio); retry: - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); folio_mark_dirty(folio); if (folio_clear_dirty_for_io(folio)) { diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c index 2156fb1fc57d..4d27eb0b32b5 100644 --- a/fs/f2fs/inline.c +++ b/fs/f2fs/inline.c @@ -34,9 +34,9 @@ bool f2fs_may_inline_data(struct inode *inode) return !f2fs_post_read_required(inode); } -static bool inode_has_blocks(struct inode *inode, struct folio *ifolio) +static bool inode_has_blocks(struct inode *inode, struct f2fs_cached_block *ientry) { - struct f2fs_inode *ri = F2FS_INODE(ifolio); + struct f2fs_inode *ri = F2FS_INODE(ientry); int i; if (F2FS_HAS_BLOCKS(inode)) @@ -49,12 +49,12 @@ static bool inode_has_blocks(struct inode *inode, struct folio *ifolio) return false; } -bool f2fs_sanity_check_inline_data(struct inode *inode, struct folio *ifolio) +bool f2fs_sanity_check_inline_data(struct inode *inode, struct f2fs_cached_block *ientry) { if (!f2fs_has_inline_data(inode)) return false; - if (inode_has_blocks(inode, ifolio)) + if (inode_has_blocks(inode, ientry)) return false; if (!support_inline_data(inode)) @@ -80,7 +80,7 @@ bool f2fs_may_inline_dentry(struct inode *inode) return true; } -void f2fs_do_read_inline_data(struct folio *folio, struct folio *ifolio) +void f2fs_do_read_inline_data(struct folio *folio, struct f2fs_cached_block *ientry) { struct inode *inode = folio->mapping->host; @@ -92,13 +92,13 @@ void f2fs_do_read_inline_data(struct folio *folio, struct folio *ifolio) folio_zero_segment(folio, MAX_INLINE_DATA(inode), folio_size(folio)); /* Copy the whole inline data block */ - memcpy_to_folio(folio, 0, inline_data_addr(inode, ifolio), + memcpy_to_folio(folio, 0, inline_data_addr(inode, ientry), MAX_INLINE_DATA(inode)); if (!folio_test_uptodate(folio)) folio_mark_uptodate(folio); } -void f2fs_truncate_inline_inode(struct inode *inode, struct folio *ifolio, +void f2fs_truncate_inline_inode(struct inode *inode, struct f2fs_cached_block *ientry, u64 from) { void *addr; @@ -106,11 +106,11 @@ void f2fs_truncate_inline_inode(struct inode *inode, struct folio *ifolio, if (from >= MAX_INLINE_DATA(inode)) return; - addr = inline_data_addr(inode, ifolio); + addr = inline_data_addr(inode, ientry); - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_cache_wait_writeback(ientry); memset(addr + from, 0, MAX_INLINE_DATA(inode) - from); - folio_mark_dirty(ifolio); + f2fs_mark_cache_dirty(ientry); if (from == 0) clear_inode_flag(inode, FI_DATA_EXIST); @@ -118,27 +118,27 @@ void f2fs_truncate_inline_inode(struct inode *inode, struct folio *ifolio, int f2fs_read_inline_data(struct inode *inode, struct folio *folio) { - struct folio *ifolio; + struct f2fs_cached_block *ientry; - ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino); - if (IS_ERR(ifolio)) { + ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino); + if (IS_ERR(ientry)) { folio_unlock(folio); - return PTR_ERR(ifolio); + return PTR_ERR(ientry); } if (!f2fs_has_inline_data(inode)) { - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); return -EAGAIN; } if (folio->index) folio_zero_segment(folio, 0, folio_size(folio)); else - f2fs_do_read_inline_data(folio, ifolio); + f2fs_do_read_inline_data(folio, ientry); if (!folio_test_uptodate(folio)) folio_mark_uptodate(folio); - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); folio_unlock(folio); return 0; } @@ -186,7 +186,7 @@ int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio) f2fs_bug_on(F2FS_F_SB(folio), folio_test_writeback(folio)); - f2fs_do_read_inline_data(folio, dn->inode_folio); + f2fs_do_read_inline_data(folio, dn->inode_entry); folio_mark_dirty(folio); /* clear dirty state */ @@ -197,7 +197,7 @@ int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio) fio.old_blkaddr = dn->data_blkaddr; set_inode_flag(dn->inode, FI_HOT_DATA); f2fs_outplace_write_data(dn, &fio); - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); if (dirty) { inode_dec_dirty_pages(dn->inode); f2fs_remove_dirty_inode(dn->inode); @@ -207,8 +207,8 @@ int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio) set_inode_flag(dn->inode, FI_APPEND_WRITE); /* clear inline data and flag after data writeback */ - f2fs_truncate_inline_inode(dn->inode, dn->inode_folio, 0); - folio_clear_f2fs_inline(dn->inode_folio); + f2fs_truncate_inline_inode(dn->inode, dn->inode_entry, 0); + f2fs_cache_clear_inline(dn->inode_entry); clear_out: stat_dec_inline_inode(dn->inode); clear_inode_flag(dn->inode, FI_INLINE_DATA); @@ -221,7 +221,8 @@ int f2fs_convert_inline_inode(struct inode *inode) struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct dnode_of_data dn; struct f2fs_lock_context lc; - struct folio *ifolio, *folio; + struct f2fs_cached_block *ientry; + struct folio *folio; int err = 0; if (f2fs_hw_is_readonly(sbi) || f2fs_readonly(sbi->sb)) @@ -240,13 +241,13 @@ int f2fs_convert_inline_inode(struct inode *inode) f2fs_lock_op(sbi, &lc); - ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(ifolio)) { - err = PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(ientry)) { + err = PTR_ERR(ientry); goto out; } - set_new_dnode(&dn, inode, ifolio, ifolio, 0); + set_new_dnode(&dn, inode, ientry, ientry, 0); if (f2fs_has_inline_data(inode)) err = f2fs_convert_inline_folio(&dn, folio); @@ -266,31 +267,31 @@ int f2fs_convert_inline_inode(struct inode *inode) int f2fs_write_inline_data(struct inode *inode, struct folio *folio) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - struct folio *ifolio; + struct f2fs_cached_block *ientry; - ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); if (!f2fs_has_inline_data(inode)) { - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); return -EAGAIN; } f2fs_bug_on(F2FS_I_SB(inode), folio->index); - f2fs_folio_wait_writeback(ifolio, NODE, true, true); - memcpy_from_folio(inline_data_addr(inode, ifolio), + f2fs_cache_wait_writeback(ientry); + memcpy_from_folio(inline_data_addr(inode, ientry), folio, 0, MAX_INLINE_DATA(inode)); - folio_mark_dirty(ifolio); + f2fs_mark_cache_dirty(ientry); f2fs_clear_page_cache_dirty_tag(folio); set_inode_flag(inode, FI_APPEND_WRITE); set_inode_flag(inode, FI_DATA_EXIST); - folio_clear_f2fs_inline(ifolio); - f2fs_folio_put(ifolio, true); + f2fs_cache_clear_inline(ientry); + f2fs_put_cache(ientry, true); return 0; } @@ -308,40 +309,41 @@ int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entr * x o -> remove data blocks, and then recover inline_data * x x -> recover data blocks */ - if (IS_INODE(cache_folio(entry))) + if (IS_INODE(entry)) ri = &CACHED_NODE(entry)->i; if (f2fs_has_inline_data(inode) && ri && (ri->i_inline & F2FS_INLINE_DATA)) { - struct folio *ifolio; + struct f2fs_cached_block *ientry; process_inline: - ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_cache_wait_writeback(ientry); - src_addr = inline_data_addr(inode, cache_folio(entry)); - dst_addr = inline_data_addr(inode, ifolio); + src_addr = inline_data_addr(inode, entry); + dst_addr = inline_data_addr(inode, ientry); memcpy(dst_addr, src_addr, MAX_INLINE_DATA(inode)); set_inode_flag(inode, FI_INLINE_DATA); set_inode_flag(inode, FI_DATA_EXIST); - folio_mark_dirty(ifolio); - f2fs_folio_put(ifolio, true); + f2fs_mark_cache_dirty(ientry); + f2fs_put_cache(ientry, true); return 1; } if (f2fs_has_inline_data(inode)) { - struct folio *ifolio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); - f2fs_truncate_inline_inode(inode, ifolio, 0); + struct f2fs_cached_block *ientry = f2fs_get_inode_cache(sbi, inode->i_ino); + + if (IS_ERR(ientry)) + return PTR_ERR(ientry); + f2fs_truncate_inline_inode(inode, ientry, 0); stat_dec_inline_inode(inode); clear_inode_flag(inode, FI_INLINE_DATA); - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); } else if (ri && (ri->i_inline & F2FS_INLINE_DATA)) { int ret; @@ -356,50 +358,50 @@ int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entr struct f2fs_dir_entry *f2fs_find_in_inline_dir(struct inode *dir, const struct f2fs_filename *fname, - struct folio **res_folio, + void **dentry_block, bool use_hash) { struct f2fs_sb_info *sbi = F2FS_SB(dir->i_sb); struct f2fs_dir_entry *de; struct f2fs_dentry_ptr d; - struct folio *ifolio; + struct f2fs_cached_block *ientry; void *inline_dentry; - ifolio = f2fs_get_inode_folio(sbi, dir->i_ino); - if (IS_ERR(ifolio)) { - *res_folio = ifolio; + ientry = f2fs_get_inode_cache(sbi, dir->i_ino); + if (IS_ERR(ientry)) { + *dentry_block = ientry; return NULL; } - inline_dentry = inline_data_addr(dir, ifolio); + inline_dentry = inline_data_addr(dir, ientry); make_dentry_ptr_inline(dir, &d, inline_dentry); de = f2fs_find_target_dentry(&d, fname, NULL, use_hash); - folio_unlock(ifolio); + f2fs_unlock_cache(ientry); if (IS_ERR(de)) { - *res_folio = ERR_CAST(de); + *dentry_block = ERR_CAST(de); de = NULL; } if (de) - *res_folio = ifolio; + *dentry_block = f2fs_cache_make_dentry_block(ientry); else - f2fs_folio_put(ifolio, false); + f2fs_put_cache(ientry, false); return de; } int f2fs_make_empty_inline_dir(struct inode *inode, struct inode *parent, - struct folio *ifolio) + struct f2fs_cached_block *ientry) { struct f2fs_dentry_ptr d; void *inline_dentry; - inline_dentry = inline_data_addr(inode, ifolio); + inline_dentry = inline_data_addr(inode, ientry); make_dentry_ptr_inline(inode, &d, inline_dentry); f2fs_do_make_empty_dir(inode, parent, &d); - folio_mark_dirty(ifolio); + f2fs_mark_cache_dirty(ientry); /* update i_size to MAX_INLINE_DATA */ if (i_size_read(inode) < MAX_INLINE_DATA(inode)) @@ -411,8 +413,9 @@ int f2fs_make_empty_inline_dir(struct inode *inode, struct inode *parent, * NOTE: ipage is grabbed by caller, but if any error occurs, we should * release ipage in this function. */ -static int f2fs_move_inline_dirents(struct inode *dir, struct folio *ifolio, - void *inline_dentry) +static int f2fs_move_inline_dirents(struct inode *dir, + struct f2fs_cached_block *ientry, + void *inline_dentry) { struct folio *folio; struct dnode_of_data dn; @@ -422,11 +425,11 @@ static int f2fs_move_inline_dirents(struct inode *dir, struct folio *ifolio, folio = f2fs_grab_cache_folio(dir->i_mapping, 0, true); if (IS_ERR(folio)) { - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); return PTR_ERR(folio); } - set_new_dnode(&dn, dir, ifolio, NULL, 0); + set_new_dnode(&dn, dir, ientry, NULL, 0); err = f2fs_reserve_block(&dn, 0); if (err) goto out; @@ -442,7 +445,7 @@ static int f2fs_move_inline_dirents(struct inode *dir, struct folio *ifolio, goto out; } - f2fs_folio_wait_writeback(folio, DATA, true, true); + f2fs_folio_wait_writeback(folio, true, true); dentry_blk = folio_address(folio); @@ -465,7 +468,7 @@ static int f2fs_move_inline_dirents(struct inode *dir, struct folio *ifolio, folio_mark_dirty(folio); /* clear inline dir and flag after data writeback */ - f2fs_truncate_inline_inode(dir, ifolio, 0); + f2fs_truncate_inline_inode(dir, ientry, 0); stat_dec_inline_dir(dir); clear_inode_flag(dir, FI_INLINE_DENTRY); @@ -545,8 +548,8 @@ static int f2fs_add_inline_entries(struct inode *dir, void *inline_dentry) return err; } -static int f2fs_move_rehashed_dirents(struct inode *dir, struct folio *ifolio, - void *inline_dentry) +static int f2fs_move_rehashed_dirents(struct inode *dir, + struct f2fs_cached_block *ientry, void *inline_dentry) { void *backup_dentry; int err; @@ -554,20 +557,20 @@ static int f2fs_move_rehashed_dirents(struct inode *dir, struct folio *ifolio, backup_dentry = f2fs_kmalloc(F2FS_I_SB(dir), MAX_INLINE_DATA(dir), GFP_F2FS_ZERO); if (!backup_dentry) { - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); return -ENOMEM; } memcpy(backup_dentry, inline_dentry, MAX_INLINE_DATA(dir)); - f2fs_truncate_inline_inode(dir, ifolio, 0); + f2fs_truncate_inline_inode(dir, ientry, 0); - folio_unlock(ifolio); + f2fs_unlock_cache(ientry); err = f2fs_add_inline_entries(dir, backup_dentry); if (err) goto recover; - folio_lock(ifolio); + f2fs_lock_cache(ientry); stat_dec_inline_dir(dir); clear_inode_flag(dir, FI_INLINE_DENTRY); @@ -583,31 +586,31 @@ static int f2fs_move_rehashed_dirents(struct inode *dir, struct folio *ifolio, kfree(backup_dentry); return 0; recover: - folio_lock(ifolio); - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_lock_cache(ientry); + f2fs_cache_wait_writeback(ientry); memcpy(inline_dentry, backup_dentry, MAX_INLINE_DATA(dir)); f2fs_i_depth_write(dir, 0); f2fs_i_size_write(dir, MAX_INLINE_DATA(dir)); - folio_mark_dirty(ifolio); - f2fs_folio_put(ifolio, true); + f2fs_mark_cache_dirty(ientry); + f2fs_put_cache(ientry, true); kfree(backup_dentry); return err; } -static int do_convert_inline_dir(struct inode *dir, struct folio *ifolio, - void *inline_dentry) +static int do_convert_inline_dir(struct inode *dir, + struct f2fs_cached_block *ientry, void *inline_dentry) { if (!F2FS_I(dir)->i_dir_level) - return f2fs_move_inline_dirents(dir, ifolio, inline_dentry); + return f2fs_move_inline_dirents(dir, ientry, inline_dentry); else - return f2fs_move_rehashed_dirents(dir, ifolio, inline_dentry); + return f2fs_move_rehashed_dirents(dir, ientry, inline_dentry); } int f2fs_try_convert_inline_dir(struct inode *dir, struct dentry *dentry) { struct f2fs_sb_info *sbi = F2FS_I_SB(dir); - struct folio *ifolio; + struct f2fs_cached_block *ientry; struct f2fs_filename fname; struct f2fs_lock_context lc; void *inline_dentry = NULL; @@ -622,22 +625,22 @@ int f2fs_try_convert_inline_dir(struct inode *dir, struct dentry *dentry) if (err) goto out; - ifolio = f2fs_get_inode_folio(sbi, dir->i_ino); - if (IS_ERR(ifolio)) { - err = PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, dir->i_ino); + if (IS_ERR(ientry)) { + err = PTR_ERR(ientry); goto out_fname; } - if (f2fs_has_enough_room(dir, ifolio, &fname)) { - f2fs_folio_put(ifolio, true); + if (f2fs_has_enough_room(dir, ientry, &fname)) { + f2fs_put_cache(ientry, true); goto out_fname; } - inline_dentry = inline_data_addr(dir, ifolio); + inline_dentry = inline_data_addr(dir, ientry); - err = do_convert_inline_dir(dir, ifolio, inline_dentry); + err = do_convert_inline_dir(dir, ientry, inline_dentry); if (!err) - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); out_fname: f2fs_free_filename(&fname); out: @@ -649,24 +652,24 @@ int f2fs_add_inline_entry(struct inode *dir, const struct f2fs_filename *fname, struct inode *inode, nid_t ino, umode_t mode) { struct f2fs_sb_info *sbi = F2FS_I_SB(dir); - struct folio *ifolio; + struct f2fs_cached_block *ientry; unsigned int bit_pos; void *inline_dentry = NULL; struct f2fs_dentry_ptr d; int slots = GET_DENTRY_SLOTS(fname->disk_name.len); - struct folio *folio = NULL; + struct f2fs_cached_block *nentry = NULL; int err = 0; - ifolio = f2fs_get_inode_folio(sbi, dir->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(sbi, dir->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); - inline_dentry = inline_data_addr(dir, ifolio); + inline_dentry = inline_data_addr(dir, ientry); make_dentry_ptr_inline(dir, &d, inline_dentry); bit_pos = f2fs_room_for_filename(d.bitmap, slots, d.max); if (bit_pos >= d.max) { - err = do_convert_inline_dir(dir, ifolio, inline_dentry); + err = do_convert_inline_dir(dir, ientry, inline_dentry); if (err) return err; err = -EAGAIN; @@ -676,19 +679,19 @@ int f2fs_add_inline_entry(struct inode *dir, const struct f2fs_filename *fname, if (inode) { f2fs_down_write_nested(&F2FS_I(inode)->i_sem, SINGLE_DEPTH_NESTING); - folio = f2fs_init_inode_metadata(inode, dir, fname, ifolio); - if (IS_ERR(folio)) { - err = PTR_ERR(folio); + nentry = f2fs_init_inode_metadata(inode, dir, fname, ientry); + if (IS_ERR(nentry)) { + err = PTR_ERR(nentry); goto fail; } } - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_cache_wait_writeback(ientry); f2fs_update_dentry(ino, mode, &d, &fname->disk_name, fname->hash, bit_pos); - folio_mark_dirty(ifolio); + f2fs_mark_cache_dirty(ientry); /* we don't need to mark_inode_dirty now */ if (inode) { @@ -696,9 +699,9 @@ int f2fs_add_inline_entry(struct inode *dir, const struct f2fs_filename *fname, /* synchronize inode page's data from inode cache */ if (is_inode_flag_set(inode, FI_NEW_INODE)) - f2fs_update_inode(inode, folio); + f2fs_update_inode(inode, nentry); - f2fs_folio_put(folio, true); + f2fs_put_cache(nentry, true); } f2fs_update_parent_metadata(dir, inode, 0); @@ -706,12 +709,12 @@ int f2fs_add_inline_entry(struct inode *dir, const struct f2fs_filename *fname, if (inode) f2fs_up_write(&F2FS_I(inode)->i_sem); out: - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); return err; } void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry, - struct folio *folio, struct inode *dir, struct inode *inode) + struct f2fs_cached_block *ientry, struct inode *dir, struct inode *inode) { struct f2fs_dentry_ptr d; void *inline_dentry; @@ -719,18 +722,18 @@ void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry, unsigned int bit_pos; int i; - folio_lock(folio); - f2fs_folio_wait_writeback(folio, NODE, true, true); + f2fs_lock_cache(ientry); + f2fs_cache_wait_writeback(ientry); - inline_dentry = inline_data_addr(dir, folio); + inline_dentry = inline_data_addr(dir, ientry); make_dentry_ptr_inline(dir, &d, inline_dentry); bit_pos = dentry - d.dentry; for (i = 0; i < slots; i++) __clear_bit_le(bit_pos + i, d.bitmap); - folio_mark_dirty(folio); - f2fs_folio_put(folio, true); + f2fs_mark_cache_dirty(ientry); + f2fs_put_cache(ientry, true); inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir)); f2fs_mark_inode_dirty_sync(dir, true); @@ -742,21 +745,21 @@ void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry, bool f2fs_empty_inline_dir(struct inode *dir) { struct f2fs_sb_info *sbi = F2FS_I_SB(dir); - struct folio *ifolio; + struct f2fs_cached_block *ientry; unsigned int bit_pos = 2; void *inline_dentry; struct f2fs_dentry_ptr d; - ifolio = f2fs_get_inode_folio(sbi, dir->i_ino); - if (IS_ERR(ifolio)) + ientry = f2fs_get_inode_cache(sbi, dir->i_ino); + if (IS_ERR(ientry)) return false; - inline_dentry = inline_data_addr(dir, ifolio); + inline_dentry = inline_data_addr(dir, ientry); make_dentry_ptr_inline(dir, &d, inline_dentry); bit_pos = find_next_bit_le(d.bitmap, d.max, bit_pos); - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); if (bit_pos < d.max) return false; @@ -768,7 +771,7 @@ int f2fs_read_inline_dir(struct file *file, struct dir_context *ctx, struct fscrypt_str *fstr) { struct inode *inode = file_inode(file); - struct folio *ifolio = NULL; + struct f2fs_cached_block *ientry = NULL; struct f2fs_dentry_ptr d; void *inline_dentry = NULL; int err; @@ -778,17 +781,17 @@ int f2fs_read_inline_dir(struct file *file, struct dir_context *ctx, if (ctx->pos == d.max) return 0; - ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); /* * f2fs_readdir was protected by inode.i_rwsem, it is safe to access * ipage without page's lock held. */ - folio_unlock(ifolio); + f2fs_unlock_cache(ientry); - inline_dentry = inline_data_addr(inode, ifolio); + inline_dentry = inline_data_addr(inode, ientry); make_dentry_ptr_inline(inode, &d, inline_dentry); @@ -796,7 +799,7 @@ int f2fs_read_inline_dir(struct file *file, struct dir_context *ctx, if (!err) ctx->pos = d.max; - f2fs_folio_put(ifolio, false); + f2fs_put_cache(ientry, false); return err < 0 ? err : 0; } @@ -807,12 +810,12 @@ int f2fs_inline_data_fiemap(struct inode *inode, __u32 flags = FIEMAP_EXTENT_DATA_INLINE | FIEMAP_EXTENT_NOT_ALIGNED | FIEMAP_EXTENT_LAST; struct node_info ni; - struct folio *ifolio; + struct f2fs_cached_block *ientry; int err = 0; - ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); if ((S_ISREG(inode->i_mode) || S_ISLNK(inode->i_mode)) && !f2fs_has_inline_data(inode)) { @@ -826,13 +829,13 @@ int f2fs_inline_data_fiemap(struct inode *inode, } if (fieinfo->fi_flags & FIEMAP_FLAG_SYNC) { - err = f2fs_write_single_node_folio(ifolio, true, false, FS_NODE_IO); + err = f2fs_write_node_cache(ientry, true, false, FS_NODE_IO); if (err) return err; - ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); + f2fs_cache_wait_writeback(ientry); } ilen = min_t(size_t, MAX_INLINE_DATA(inode), i_size_read(inode)); if (start >= ilen) @@ -847,8 +850,8 @@ int f2fs_inline_data_fiemap(struct inode *inode, if (__is_valid_data_blkaddr(ni.blk_addr)) { byteaddr = (__u64)ni.blk_addr << inode->i_sb->s_blocksize_bits; - byteaddr += (char *)inline_data_addr(inode, ifolio) - - (char *)F2FS_INODE(ifolio); + byteaddr += (char *)inline_data_addr(inode, ientry) - + (char *)F2FS_INODE(ientry); } else { f2fs_bug_on(F2FS_I_SB(inode), ni.blk_addr != NEW_ADDR); flags |= FIEMAP_EXTENT_DELALLOC | FIEMAP_EXTENT_UNKNOWN; @@ -856,6 +859,6 @@ int f2fs_inline_data_fiemap(struct inode *inode, err = fiemap_fill_next_extent(fieinfo, start, byteaddr, ilen, flags); trace_f2fs_fiemap(inode, start, byteaddr, ilen, flags, err); out: - f2fs_folio_put(ifolio, true); + f2fs_put_cache(ientry, true); return err; } diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c index c533da4d4d70..421788a9ea24 100644 --- a/fs/f2fs/inode.c +++ b/fs/f2fs/inode.c @@ -82,9 +82,9 @@ void f2fs_set_inode_flags(struct inode *inode) S_ENCRYPTED|S_VERITY|S_CASEFOLD); } -static void __get_inode_rdev(struct inode *inode, struct folio *node_folio) +static void __get_inode_rdev(struct inode *inode, struct f2fs_cached_block *ientry) { - __le32 *addr = get_dnode_addr(inode, node_folio); + __le32 *addr = get_dnode_addr(inode, ientry); if (S_ISCHR(inode->i_mode) || S_ISBLK(inode->i_mode) || S_ISFIFO(inode->i_mode) || S_ISSOCK(inode->i_mode)) { @@ -95,9 +95,9 @@ static void __get_inode_rdev(struct inode *inode, struct folio *node_folio) } } -static void __set_inode_rdev(struct inode *inode, struct folio *node_folio) +static void __set_inode_rdev(struct inode *inode, struct f2fs_cached_block *ientry) { - __le32 *addr = get_dnode_addr(inode, node_folio); + __le32 *addr = get_dnode_addr(inode, ientry); if (S_ISCHR(inode->i_mode) || S_ISBLK(inode->i_mode)) { if (old_valid_dev(inode->i_rdev)) { @@ -111,34 +111,33 @@ static void __set_inode_rdev(struct inode *inode, struct folio *node_folio) } } -static void __recover_inline_status(struct inode *inode, struct folio *ifolio) +static void __recover_inline_status(struct inode *inode, struct f2fs_cached_block *ientry) { - void *inline_data = inline_data_addr(inode, ifolio); + void *inline_data = inline_data_addr(inode, ientry); __le32 *start = inline_data; __le32 *end = start + MAX_INLINE_DATA(inode) / sizeof(__le32); while (start < end) { if (*start++) { - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_cache_wait_writeback(ientry); set_inode_flag(inode, FI_DATA_EXIST); - set_raw_inline(inode, F2FS_INODE(ifolio)); - folio_mark_dirty(ifolio); + set_raw_inline(inode, F2FS_INODE(ientry)); + f2fs_mark_cache_dirty(ientry); return; } } return; } -static -bool f2fs_enable_inode_chksum(struct f2fs_sb_info *sbi, struct folio *folio) +static bool f2fs_enable_inode_chksum(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) { - struct f2fs_inode *ri = &F2FS_NODE(folio)->i; + struct f2fs_inode *ri = F2FS_INODE(entry); if (!f2fs_sb_has_inode_chksum(sbi)) return false; - if (!IS_INODE(folio) || !(ri->i_inline & F2FS_EXTRA_ATTR)) + if (!IS_INODE(entry) || !(ri->i_inline & F2FS_EXTRA_ATTR)) return false; if (!F2FS_FITS_IN_INODE(ri, le16_to_cpu(ri->i_extra_isize), @@ -148,10 +147,10 @@ bool f2fs_enable_inode_chksum(struct f2fs_sb_info *sbi, struct folio *folio) return true; } -static __u32 f2fs_inode_chksum(struct f2fs_sb_info *sbi, struct folio *folio) +static __u32 f2fs_inode_chksum(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) { - struct f2fs_node *node = F2FS_NODE(folio); - struct f2fs_inode *ri = &node->i; + struct f2fs_node *node = CACHED_NODE(entry); + struct f2fs_inode *ri = F2FS_INODE(entry); __le32 ino = node->footer.ino; __le32 gen = ri->i_generation; __u32 chksum, chksum_seed; @@ -170,7 +169,7 @@ static __u32 f2fs_inode_chksum(struct f2fs_sb_info *sbi, struct folio *folio) return chksum; } -bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct folio *folio) +bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) { struct f2fs_inode *ri; __u32 provided, calculated; @@ -178,35 +177,33 @@ bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct folio *folio) if (unlikely(is_sbi_flag_set(sbi, SBI_IS_SHUTDOWN))) return true; -#ifdef CONFIG_F2FS_CHECK_FS - if (!f2fs_enable_inode_chksum(sbi, folio)) -#else - if (!f2fs_enable_inode_chksum(sbi, folio) || - folio_test_dirty(folio) || - folio_test_writeback(folio)) -#endif + if (!f2fs_enable_inode_chksum(sbi, entry)) return true; +#ifndef CONFIG_F2FS_CHECK_FS + if (f2fs_cache_test_dirty(entry) || f2fs_cache_test_writeback(entry)) + return true; +#endif - ri = &F2FS_NODE(folio)->i; + ri = F2FS_INODE(entry); provided = le32_to_cpu(ri->i_inode_checksum); - calculated = f2fs_inode_chksum(sbi, folio); + calculated = f2fs_inode_chksum(sbi, entry); if (provided != calculated) - f2fs_warn(sbi, "checksum invalid, nid = %lu, ino_of_node = %x, %x vs. %x", - folio->index, ino_of_node(folio), + f2fs_warn(sbi, "checksum invalid, nid = %lu, ino_of_node = %u, %x vs. %x", + entry->index, ino_of_node(entry), provided, calculated); return provided == calculated; } -void f2fs_inode_chksum_set(struct f2fs_sb_info *sbi, struct folio *folio) +void f2fs_inode_chksum_set(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) { - struct f2fs_inode *ri = &F2FS_NODE(folio)->i; + struct f2fs_inode *ri = F2FS_INODE(entry); - if (!f2fs_enable_inode_chksum(sbi, folio)) + if (!f2fs_enable_inode_chksum(sbi, entry)) return; - ri->i_inode_checksum = cpu_to_le32(f2fs_inode_chksum(sbi, folio)); + ri->i_inode_checksum = cpu_to_le32(f2fs_inode_chksum(sbi, entry)); } static bool sanity_check_compress_inode(struct inode *inode, @@ -281,28 +278,29 @@ static bool sanity_check_compress_inode(struct inode *inode, return false; } -static bool sanity_check_inode(struct inode *inode, struct folio *node_folio) +static bool sanity_check_inode(struct inode *inode, + struct f2fs_cached_block *node_entry) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct f2fs_inode_info *fi = F2FS_I(inode); - struct f2fs_inode *ri = F2FS_INODE(node_folio); + struct f2fs_inode *ri = F2FS_INODE(node_entry); unsigned long long iblocks; - iblocks = le64_to_cpu(F2FS_INODE(node_folio)->i_blocks); + iblocks = le64_to_cpu(ri->i_blocks); if (!iblocks) { f2fs_warn(sbi, "%s: corrupted inode i_blocks i_ino=%llx iblocks=%llu, run fsck to fix.", __func__, inode->i_ino, iblocks); return false; } - if (ino_of_node(node_folio) != nid_of_node(node_folio)) { + if (ino_of_node(node_entry) != nid_of_node(node_entry)) { f2fs_warn(sbi, "%s: corrupted inode footer i_ino=%llx, ino,nid: [%u, %u] run fsck to fix.", __func__, inode->i_ino, - ino_of_node(node_folio), nid_of_node(node_folio)); + ino_of_node(node_entry), nid_of_node(node_entry)); return false; } - if (ino_of_node(node_folio) == fi->i_xattr_nid) { + if (ino_of_node(node_entry) == fi->i_xattr_nid) { f2fs_warn(sbi, "%s: corrupted inode i_ino=%llx, xnid=%x, run fsck to fix.", __func__, inode->i_ino, fi->i_xattr_nid); return false; @@ -375,7 +373,7 @@ static bool sanity_check_inode(struct inode *inode, struct folio *node_folio) } } - if (f2fs_sanity_check_inline_data(inode, node_folio)) { + if (f2fs_sanity_check_inline_data(inode, node_entry)) { f2fs_warn(sbi, "%s: inode (ino=%llx, mode=%u) should not have inline_data, run fsck to fix", __func__, inode->i_ino, inode->i_mode); return false; @@ -428,7 +426,7 @@ static int do_read_inode(struct inode *inode) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct f2fs_inode_info *fi = F2FS_I(inode); - struct folio *node_folio; + struct f2fs_cached_block *node_entry; struct f2fs_inode *ri; projid_t i_projid; @@ -436,11 +434,11 @@ static int do_read_inode(struct inode *inode) if (f2fs_check_nid_range(sbi, inode->i_ino)) return -EINVAL; - node_folio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(node_folio)) - return PTR_ERR(node_folio); + node_entry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(node_entry)) + return PTR_ERR(node_entry); - ri = F2FS_INODE(node_folio); + ri = F2FS_INODE(node_entry); inode->i_mode = le16_to_cpu(ri->i_mode); i_uid_write(inode, le32_to_cpu(ri->i_uid)); @@ -490,8 +488,8 @@ static int do_read_inode(struct inode *inode) fi->i_inline_xattr_size = 0; } - if (!sanity_check_inode(inode, node_folio)) { - f2fs_folio_put(node_folio, true); + if (!sanity_check_inode(inode, node_entry)) { + f2fs_put_cache(node_entry, true); set_sbi_flag(sbi, SBI_NEED_FSCK); f2fs_handle_error(sbi, ERROR_CORRUPTED_INODE); fserror_report_file_metadata(inode, -EFSCORRUPTED, GFP_NOFS); @@ -500,17 +498,17 @@ static int do_read_inode(struct inode *inode) /* check data exist */ if (f2fs_has_inline_data(inode) && !f2fs_exist_data(inode)) - __recover_inline_status(inode, node_folio); + __recover_inline_status(inode, node_entry); /* try to recover cold bit for non-dir inode */ - if (!S_ISDIR(inode->i_mode) && !is_cold_node(node_folio)) { - f2fs_folio_wait_writeback(node_folio, NODE, true, true); - set_cold_node(node_folio, false); - folio_mark_dirty(node_folio); + if (!S_ISDIR(inode->i_mode) && !is_cold_node(node_entry)) { + f2fs_cache_wait_writeback(node_entry); + set_cold_node(node_entry, false); + f2fs_mark_cache_dirty(node_entry); } /* get rdev by using inline_info */ - __get_inode_rdev(inode, node_folio); + __get_inode_rdev(inode, node_entry); if (!f2fs_need_inode_block_update(sbi, inode->i_ino)) fi->last_disk_size = inode->i_size; @@ -553,18 +551,18 @@ static int do_read_inode(struct inode *inode) init_idisk_time(inode); - if (!sanity_check_extent_cache(inode, node_folio)) { - f2fs_folio_put(node_folio, true); + if (!sanity_check_extent_cache(inode, node_entry)) { + f2fs_put_cache(node_entry, true); f2fs_handle_error(sbi, ERROR_CORRUPTED_INODE); fserror_report_file_metadata(inode, -EFSCORRUPTED, GFP_NOFS); return -EFSCORRUPTED; } /* Need all the flag bits */ - f2fs_init_read_extent_tree(inode, node_folio); + f2fs_init_read_extent_tree(inode, node_entry); f2fs_init_age_extent_tree(inode); - f2fs_folio_put(node_folio, true); + f2fs_put_cache(node_entry, true); stat_inc_inline_xattr(inode); stat_inc_inline_inode(inode); @@ -577,8 +575,6 @@ static int do_read_inode(struct inode *inode) static bool is_meta_ino(struct f2fs_sb_info *sbi, unsigned int ino) { - if (ino == F2FS_NODE_INO(sbi)) - return true; #ifdef CONFIG_F2FS_FS_COMPRESSION if (test_opt(sbi, COMPRESS_CACHE) && ino == F2FS_COMPRESS_INO(sbi)) return true; @@ -621,10 +617,7 @@ struct inode *f2fs_iget(struct super_block *sb, unsigned long ino) make_now: f2fs_set_inode_flags(inode); - if (ino == F2FS_NODE_INO(sbi)) { - inode->i_mapping->a_ops = &f2fs_node_aops; - mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS); - } else if (ino == F2FS_COMPRESS_INO(sbi)) { + if (ino == F2FS_COMPRESS_INO(sbi)) { #ifdef CONFIG_F2FS_FS_COMPRESSION inode->i_mapping->a_ops = &f2fs_compress_aops; /* @@ -691,18 +684,19 @@ struct inode *f2fs_iget_retry(struct super_block *sb, unsigned long ino) return inode; } -void f2fs_update_inode(struct inode *inode, struct folio *node_folio) +void f2fs_update_inode(struct inode *inode, + struct f2fs_cached_block *node_entry) { struct f2fs_inode_info *fi = F2FS_I(inode); struct f2fs_inode *ri; struct extent_tree *et = fi->extent_tree[EX_READ]; - f2fs_folio_wait_writeback(node_folio, NODE, true, true); - folio_mark_dirty(node_folio); + f2fs_cache_wait_writeback(node_entry); + f2fs_mark_cache_dirty(node_entry); f2fs_inode_synced(inode); - ri = F2FS_INODE(node_folio); + ri = F2FS_INODE(node_entry); ri->i_mode = cpu_to_le16(inode->i_mode); ri->i_advise = fi->i_advise; @@ -777,27 +771,27 @@ void f2fs_update_inode(struct inode *inode, struct folio *node_folio) } } - __set_inode_rdev(inode, node_folio); + __set_inode_rdev(inode, node_entry); /* deleted inode */ if (inode->i_nlink == 0) - folio_clear_f2fs_inline(node_folio); + f2fs_cache_clear_inline(node_entry); init_idisk_time(inode); #ifdef CONFIG_F2FS_CHECK_FS - f2fs_inode_chksum_set(F2FS_I_SB(inode), node_folio); + f2fs_inode_chksum_set(F2FS_I_SB(inode), node_entry); #endif } void f2fs_update_inode_page(struct inode *inode) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - struct folio *node_folio; + struct f2fs_cached_block *node_entry; int count = 0; retry: - node_folio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(node_folio)) { - int err = PTR_ERR(node_folio); + node_entry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(node_entry)) { + int err = PTR_ERR(node_entry); /* The node block was truncated. */ if (err == -ENOENT) @@ -813,17 +807,14 @@ void f2fs_update_inode_page(struct inode *inode) f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_UPDATE_INODE); return; } - f2fs_update_inode(inode, node_folio); - f2fs_folio_put(node_folio, true); + f2fs_update_inode(inode, node_entry); + f2fs_put_cache(node_entry, true); } int f2fs_write_inode(struct inode *inode, struct writeback_control *wbc) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - if (inode->i_ino == F2FS_NODE_INO(sbi)) - return 0; - /* * atime could be updated without dirtying f2fs inode in lazytime mode */ @@ -914,8 +905,7 @@ static bool f2fs_pre_evict_inode(struct inode *inode) test_opt(sbi, COMPRESS_CACHE) && f2fs_compressed_file(inode)) f2fs_invalidate_compress_pages(sbi, inode->i_ino); - if (inode->i_ino == F2FS_NODE_INO(sbi) || - inode->i_ino == F2FS_COMPRESS_INO(sbi)) + if (inode->i_ino == F2FS_COMPRESS_INO(sbi)) return true; f2fs_bug_on(sbi, get_dirty_pages(inode)); @@ -1041,10 +1031,9 @@ static void f2fs_post_evict_inode(struct inode *inode) /* for the case f2fs_new_inode() was failed, .i_ino is zero, skip it */ if (inode->i_ino) - invalidate_mapping_pages(NODE_MAPPING(sbi), inode->i_ino, - inode->i_ino); + f2fs_invalidate_node_cache(sbi, inode->i_ino); if (xnid) - invalidate_mapping_pages(NODE_MAPPING(sbi), xnid, xnid); + f2fs_invalidate_node_cache(sbi, xnid); if (!inode->i_nlink) goto skip_record; diff --git a/fs/f2fs/namei.c b/fs/f2fs/namei.c index 37897f4321c0..43760f71e622 100644 --- a/fs/f2fs/namei.c +++ b/fs/f2fs/namei.c @@ -473,12 +473,13 @@ static int f2fs_link(struct dentry *old_dentry, struct inode *dir, struct dentry *f2fs_get_parent(struct dentry *child) { - struct folio *folio; - unsigned long ino = f2fs_inode_by_name(d_inode(child), &dotdot_name, &folio); + void *dentry_block = NULL; + unsigned long ino = f2fs_inode_by_name(d_inode(child), + &dotdot_name, &dentry_block); if (!ino) { - if (IS_ERR(folio)) - return ERR_CAST(folio); + if (IS_ERR(dentry_block)) + return ERR_CAST(dentry_block); return ERR_PTR(-ENOENT); } return d_obtain_alias(f2fs_iget(child->d_sb, ino)); @@ -489,7 +490,7 @@ static struct dentry *f2fs_lookup(struct inode *dir, struct dentry *dentry, { struct inode *inode = NULL; struct f2fs_dir_entry *de; - struct folio *folio; + void *dentry_block = NULL; struct dentry *new; nid_t ino = -1; int err = 0; @@ -507,12 +508,12 @@ static struct dentry *f2fs_lookup(struct inode *dir, struct dentry *dentry, goto out_splice; if (err) goto out; - de = __f2fs_find_entry(dir, &fname, &folio); + de = __f2fs_find_entry(dir, &fname, &dentry_block); f2fs_free_filename(&fname); if (!de) { - if (IS_ERR(folio)) { - err = PTR_ERR(folio); + if (IS_ERR(dentry_block)) { + err = PTR_ERR(dentry_block); goto out; } err = -ENOENT; @@ -520,7 +521,7 @@ static struct dentry *f2fs_lookup(struct inode *dir, struct dentry *dentry, } ino = le32_to_cpu(de->ino); - f2fs_folio_put(folio, false); + f2fs_put_dentry_block(dentry_block, false); inode = f2fs_iget(dir->i_sb, ino); if (IS_ERR(inode)) { @@ -572,7 +573,7 @@ static int f2fs_unlink(struct inode *dir, struct dentry *dentry) struct inode *inode = d_inode(dentry); struct f2fs_dir_entry *de; struct f2fs_lock_context lc; - struct folio *folio; + void *dentry_block = NULL; int err; trace_f2fs_unlink_enter(dir, dentry); @@ -592,10 +593,10 @@ static int f2fs_unlink(struct inode *dir, struct dentry *dentry) if (err) goto out; - de = f2fs_find_entry(dir, &dentry->d_name, &folio); + de = f2fs_find_entry(dir, &dentry->d_name, &dentry_block); if (!de) { - if (IS_ERR(folio)) - err = PTR_ERR(folio); + if (IS_ERR(dentry_block)) + err = PTR_ERR(dentry_block); goto out; } @@ -615,10 +616,10 @@ static int f2fs_unlink(struct inode *dir, struct dentry *dentry) err = f2fs_acquire_orphan_inode(sbi); if (err) { f2fs_unlock_op(sbi, &lc); - f2fs_folio_put(folio, false); + f2fs_put_dentry_block(dentry_block, false); goto out; } - f2fs_delete_entry(de, folio, dir, inode); + f2fs_delete_entry(de, dentry_block, dir, inode); f2fs_unlock_op(sbi, &lc); /* VFS negative dentries are incompatible with Encoding and @@ -640,7 +641,7 @@ static int f2fs_unlink(struct inode *dir, struct dentry *dentry) corrupted: err = -EFSCORRUPTED; set_sbi_flag(sbi, SBI_NEED_FSCK); - f2fs_folio_put(folio, false); + f2fs_put_dentry_block(dentry_block, false); out: trace_f2fs_unlink_exit(inode, err); return err; @@ -959,8 +960,9 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, struct inode *old_inode = d_inode(old_dentry); struct inode *new_inode = d_inode(new_dentry); struct inode *whiteout = NULL; - struct folio *old_dir_folio = NULL; - struct folio *old_folio, *new_folio = NULL; + void *old_dir_block = NULL; + void *old_block = NULL; + void *new_block = NULL; struct f2fs_dir_entry *old_dir_entry = NULL; struct f2fs_dir_entry *old_entry; struct f2fs_dir_entry *new_entry; @@ -1024,18 +1026,18 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, } err = -ENOENT; - old_entry = f2fs_find_entry(old_dir, &old_dentry->d_name, &old_folio); + old_entry = f2fs_find_entry(old_dir, &old_dentry->d_name, &old_block); if (!old_entry) { - if (IS_ERR(old_folio)) - err = PTR_ERR(old_folio); + if (IS_ERR(old_block)) + err = PTR_ERR(old_block); goto out; } if (old_is_dir && old_dir != new_dir) { - old_dir_entry = f2fs_parent_dir(old_inode, &old_dir_folio); + old_dir_entry = f2fs_parent_dir(old_inode, &old_dir_block); if (!old_dir_entry) { - if (IS_ERR(old_dir_folio)) - err = PTR_ERR(old_dir_folio); + if (IS_ERR(old_dir_block)) + err = PTR_ERR(old_dir_block); goto out_old; } } @@ -1050,10 +1052,10 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, err = -ENOENT; new_entry = f2fs_find_entry(new_dir, &new_dentry->d_name, - &new_folio); + &new_block); if (!new_entry) { - if (IS_ERR(new_folio)) - err = PTR_ERR(new_folio); + if (IS_ERR(new_block)) + err = PTR_ERR(new_block); goto out_dir; } @@ -1065,8 +1067,8 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, if (err) goto put_out_dir; - f2fs_set_link(new_dir, new_entry, new_folio, old_inode); - new_folio = NULL; + f2fs_set_link(new_dir, new_entry, new_block, old_inode); + new_block = NULL; inode_set_ctime_current(new_inode); f2fs_down_write(&F2FS_I(new_inode)->i_sem); @@ -1105,8 +1107,8 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, inode_set_ctime_current(old_inode); f2fs_mark_inode_dirty_sync(old_inode, true); - f2fs_delete_entry(old_entry, old_folio, old_dir, NULL); - old_folio = NULL; + f2fs_delete_entry(old_entry, old_block, old_dir, NULL); + old_block = NULL; if (whiteout) { set_inode_flag(whiteout, FI_INC_LINK); @@ -1124,7 +1126,7 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, } if (old_dir_entry) - f2fs_set_link(old_inode, old_dir_entry, old_dir_folio, new_dir); + f2fs_set_link(old_inode, old_dir_entry, old_dir_block, new_dir); if (old_is_dir) f2fs_i_links_write(old_dir, false); @@ -1148,12 +1150,12 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, put_out_dir: f2fs_unlock_op(sbi, &lc); - f2fs_folio_put(new_folio, false); + f2fs_put_dentry_block(new_block, false); out_dir: if (old_dir_entry) - f2fs_folio_put(old_dir_folio, false); + f2fs_put_dentry_block(old_dir_block, false); out_old: - f2fs_folio_put(old_folio, false); + f2fs_put_dentry_block(old_block, false); out: iput(whiteout); return err; @@ -1165,8 +1167,10 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, struct f2fs_sb_info *sbi = F2FS_I_SB(old_dir); struct inode *old_inode = d_inode(old_dentry); struct inode *new_inode = d_inode(new_dentry); - struct folio *old_dir_folio, *new_dir_folio; - struct folio *old_folio, *new_folio; + void *old_dir_block = NULL; + void *new_dir_block = NULL; + void *old_block = NULL; + void *new_block = NULL; struct f2fs_dir_entry *old_dir_entry = NULL, *new_dir_entry = NULL; struct f2fs_dir_entry *old_entry, *new_entry; struct f2fs_lock_context lc; @@ -1198,17 +1202,17 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, goto out; err = -ENOENT; - old_entry = f2fs_find_entry(old_dir, &old_dentry->d_name, &old_folio); + old_entry = f2fs_find_entry(old_dir, &old_dentry->d_name, &old_block); if (!old_entry) { - if (IS_ERR(old_folio)) - err = PTR_ERR(old_folio); + if (IS_ERR(old_block)) + err = PTR_ERR(old_block); goto out; } - new_entry = f2fs_find_entry(new_dir, &new_dentry->d_name, &new_folio); + new_entry = f2fs_find_entry(new_dir, &new_dentry->d_name, &new_block); if (!new_entry) { - if (IS_ERR(new_folio)) - err = PTR_ERR(new_folio); + if (IS_ERR(new_block)) + err = PTR_ERR(new_block); goto out_old; } @@ -1216,20 +1220,20 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, if (old_dir != new_dir) { if (S_ISDIR(old_inode->i_mode)) { old_dir_entry = f2fs_parent_dir(old_inode, - &old_dir_folio); + &old_dir_block); if (!old_dir_entry) { - if (IS_ERR(old_dir_folio)) - err = PTR_ERR(old_dir_folio); + if (IS_ERR(old_dir_block)) + err = PTR_ERR(old_dir_block); goto out_new; } } if (S_ISDIR(new_inode->i_mode)) { new_dir_entry = f2fs_parent_dir(new_inode, - &new_dir_folio); + &new_dir_block); if (!new_dir_entry) { - if (IS_ERR(new_dir_folio)) - err = PTR_ERR(new_dir_folio); + if (IS_ERR(new_dir_block)) + err = PTR_ERR(new_dir_block); goto out_old_dir; } } @@ -1256,14 +1260,14 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, /* update ".." directory entry info of old dentry */ if (old_dir_entry) - f2fs_set_link(old_inode, old_dir_entry, old_dir_folio, new_dir); + f2fs_set_link(old_inode, old_dir_entry, old_dir_block, new_dir); /* update ".." directory entry info of new dentry */ if (new_dir_entry) - f2fs_set_link(new_inode, new_dir_entry, new_dir_folio, old_dir); + f2fs_set_link(new_inode, new_dir_entry, new_dir_block, old_dir); /* update directory entry info of old dir inode */ - f2fs_set_link(old_dir, old_entry, old_folio, new_inode); + f2fs_set_link(old_dir, old_entry, old_block, new_inode); f2fs_down_write(&F2FS_I(old_inode)->i_sem); if (!old_dir_entry) @@ -1282,7 +1286,7 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, f2fs_mark_inode_dirty_sync(old_dir, true); /* update directory entry info of new dir inode */ - f2fs_set_link(new_dir, new_entry, new_folio, old_inode); + f2fs_set_link(new_dir, new_entry, new_block, old_inode); f2fs_down_write(&F2FS_I(new_inode)->i_sem); if (!new_dir_entry) @@ -1317,16 +1321,16 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, return 0; out_new_dir: if (new_dir_entry) { - f2fs_folio_put(new_dir_folio, false); + f2fs_put_dentry_block(new_dir_block, false); } out_old_dir: if (old_dir_entry) { - f2fs_folio_put(old_dir_folio, false); + f2fs_put_dentry_block(old_dir_block, false); } out_new: - f2fs_folio_put(new_folio, false); + f2fs_put_dentry_block(new_block, false); out_old: - f2fs_folio_put(old_folio, false); + f2fs_put_dentry_block(old_block, false); out: return err; } diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c index 398c58fc6cad..9482c86b5506 100644 --- a/fs/f2fs/node.c +++ b/fs/f2fs/node.c @@ -13,6 +13,7 @@ #include <linux/folio_batch.h> #include <linux/swap.h> #include <linux/fserror.h> +#include <linux/freezer.h> #include "f2fs.h" #include "node.h" @@ -130,16 +131,6 @@ bool f2fs_available_free_memory(struct f2fs_sb_info *sbi, int type) return res; } -static void clear_node_folio_dirty(struct folio *folio) -{ - if (folio_test_dirty(folio)) { - f2fs_clear_page_cache_dirty_tag(folio); - folio_clear_dirty_for_io(folio); - dec_page_count(F2FS_F_SB(folio), F2FS_DIRTY_NODES); - } - folio_clear_uptodate(folio); -} - static struct f2fs_cached_block *get_current_nat_cache(struct f2fs_sb_info *sbi, nid_t nid) { @@ -328,9 +319,12 @@ static unsigned int __gang_lookup_nat_set(struct f2fs_nm_info *nm_i, start, nr); } -bool f2fs_in_warm_node_list(struct folio *folio) + + +bool f2fs_in_warm_node_list(struct f2fs_cached_block *entry) { - return is_node_folio(folio) && IS_DNODE(folio) && is_cold_node(folio); + return f2fs_is_node_cache(entry) && IS_DNODE(entry) && + is_cold_node(entry); } void f2fs_init_fsync_node_info(struct f2fs_sb_info *sbi) @@ -342,7 +336,7 @@ void f2fs_init_fsync_node_info(struct f2fs_sb_info *sbi) } static unsigned int f2fs_add_fsync_node_entry(struct f2fs_sb_info *sbi, - struct folio *folio) + struct f2fs_cached_block *entry) { struct fsync_node_entry *fn; unsigned long flags; @@ -351,8 +345,8 @@ static unsigned int f2fs_add_fsync_node_entry(struct f2fs_sb_info *sbi, fn = f2fs_kmem_cache_alloc(fsync_node_entry_slab, GFP_NOFS, true, NULL); - folio_get(folio); - fn->folio = folio; + f2fs_cache_get(entry); + fn->entry = entry; INIT_LIST_HEAD(&fn->list); spin_lock_irqsave(&sbi->fsync_node_lock, flags); @@ -365,19 +359,19 @@ static unsigned int f2fs_add_fsync_node_entry(struct f2fs_sb_info *sbi, return seq_id; } -void f2fs_del_fsync_node_entry(struct f2fs_sb_info *sbi, struct folio *folio) +void f2fs_del_fsync_node_entry(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) { struct fsync_node_entry *fn; unsigned long flags; spin_lock_irqsave(&sbi->fsync_node_lock, flags); list_for_each_entry(fn, &sbi->fsync_node_list, list) { - if (fn->folio == folio) { + if (fn->entry == entry) { list_del(&fn->list); sbi->fsync_node_num--; spin_unlock_irqrestore(&sbi->fsync_node_lock, flags); + f2fs_put_cache(fn->entry, false); kmem_cache_free(fsync_node_entry_slab, fn); - folio_put(folio); return; } } @@ -670,9 +664,9 @@ int f2fs_get_node_info(struct f2fs_sb_info *sbi, nid_t nid, /* * readahead MAX_RA_NODE number of node pages. */ -static void f2fs_ra_node_pages(struct folio *parent, int start, int n) +static void f2fs_ra_node_pages(struct f2fs_cached_block *parent, int start, int n) { - struct f2fs_sb_info *sbi = F2FS_F_SB(parent); + struct f2fs_sb_info *sbi = parent->cache->sbi; struct blk_plug plug; int i, end; nid_t nid; @@ -801,7 +795,8 @@ static int get_node_path(struct inode *inode, long block, return level; } -static struct folio *f2fs_get_node_folio_ra(struct folio *parent, int start); +static struct f2fs_cached_block *f2fs_get_node_cache_ra( + struct f2fs_cached_block *parent, int start); /* * Caller should call f2fs_put_dnode(dn). @@ -811,8 +806,8 @@ static struct folio *f2fs_get_node_folio_ra(struct folio *parent, int start); int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode) { struct f2fs_sb_info *sbi = F2FS_I_SB(dn->inode); - struct folio *nfolio[4]; - struct folio *parent = NULL; + struct f2fs_cached_block *nentry[4]; + struct f2fs_cached_block *parent = NULL; int offset[4]; unsigned int noffset[4]; nid_t nids[4]; @@ -825,26 +820,26 @@ int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode) nids[0] = dn->inode->i_ino; - if (!dn->inode_folio) { - nfolio[0] = f2fs_get_inode_folio(sbi, nids[0]); - if (IS_ERR(nfolio[0])) - return PTR_ERR(nfolio[0]); + if (!dn->inode_entry) { + nentry[0] = f2fs_get_inode_cache(sbi, nids[0]); + if (IS_ERR(nentry[0])) + return PTR_ERR(nentry[0]); } else { - nfolio[0] = dn->inode_folio; + nentry[0] = dn->inode_entry; } /* if inline_data is set, should not report any block indices */ if (f2fs_has_inline_data(dn->inode) && index) { err = -ENOENT; - f2fs_folio_put(nfolio[0], true); + f2fs_put_cache(nentry[0], true); goto release_out; } - parent = nfolio[0]; + parent = nentry[0]; if (level != 0) nids[1] = get_nid(parent, offset[0], true); - dn->inode_folio = nfolio[0]; - dn->inode_folio_locked = true; + dn->inode_entry = nentry[0]; + dn->inode_entry_locked = true; /* get indirect or direct nodes */ for (i = 1; i <= level; i++) { @@ -868,10 +863,10 @@ int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode) } dn->nid = nids[i]; - nfolio[i] = f2fs_new_node_folio(dn, noffset[i]); - if (IS_ERR(nfolio[i])) { + nentry[i] = f2fs_new_node_cache(dn, noffset[i]); + if (IS_ERR(nentry[i])) { f2fs_alloc_nid_failed(sbi, nids[i]); - err = PTR_ERR(nfolio[i]); + err = PTR_ERR(nentry[i]); goto release_pages; } @@ -879,37 +874,37 @@ int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode) f2fs_alloc_nid_done(sbi, nids[i]); done = true; } else if (mode == LOOKUP_NODE_RA && i == level && level > 1) { - nfolio[i] = f2fs_get_node_folio_ra(parent, offset[i - 1]); - if (IS_ERR(nfolio[i])) { - err = PTR_ERR(nfolio[i]); + nentry[i] = f2fs_get_node_cache_ra(parent, offset[i - 1]); + if (IS_ERR(nentry[i])) { + err = PTR_ERR(nentry[i]); goto release_pages; } done = true; } if (i == 1) { - dn->inode_folio_locked = false; - folio_unlock(parent); + dn->inode_entry_locked = false; + f2fs_unlock_cache(parent); } else { - f2fs_folio_put(parent, true); + f2fs_put_cache(parent, true); } if (!done) { - nfolio[i] = f2fs_get_node_folio(sbi, nids[i], + nentry[i] = f2fs_get_node_cache(sbi, nids[i], NODE_TYPE_NON_INODE); - if (IS_ERR(nfolio[i])) { - err = PTR_ERR(nfolio[i]); - f2fs_folio_put(nfolio[0], false); + if (IS_ERR(nentry[i])) { + err = PTR_ERR(nentry[i]); + f2fs_put_cache(nentry[0], false); goto release_out; } } if (i < level) { - parent = nfolio[i]; + parent = nentry[i]; nids[i + 1] = get_nid(parent, offset[i], false); } } dn->nid = nids[level]; dn->ofs_in_node = offset[level]; - dn->node_folio = nfolio[level]; + dn->node_entry = nentry[level]; dn->data_blkaddr = f2fs_data_blkaddr(dn); if (is_inode_flag_set(dn->inode, FI_COMPRESSED_FILE) && @@ -930,9 +925,9 @@ int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode) if (!c_len) goto out; - blkaddr = data_blkaddr(dn->inode, dn->node_folio, ofs_in_node); + blkaddr = data_blkaddr(dn->inode, dn->node_entry, ofs_in_node); if (blkaddr == COMPRESS_ADDR) - blkaddr = data_blkaddr(dn->inode, dn->node_folio, + blkaddr = data_blkaddr(dn->inode, dn->node_entry, ofs_in_node + 1); f2fs_update_read_extent_tree_range_compressed(dn->inode, @@ -942,12 +937,12 @@ int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode) return 0; release_pages: - f2fs_folio_put(parent, true); + f2fs_put_cache(parent, true); if (i > 1) - f2fs_folio_put(nfolio[0], false); + f2fs_put_cache(nentry[0], false); release_out: - dn->inode_folio = NULL; - dn->node_folio = NULL; + dn->inode_entry = NULL; + dn->node_entry = NULL; if (err == -ENOENT) { dn->cur_level = i; dn->max_level = level; @@ -988,16 +983,15 @@ static int truncate_node(struct dnode_of_data *dn) f2fs_inode_synced(dn->inode); } - clear_node_folio_dirty(dn->node_folio); + f2fs_drop_cache_dirty(dn->node_entry); set_sbi_flag(sbi, SBI_IS_DIRTY); - index = dn->node_folio->index; - f2fs_folio_put(dn->node_folio, true); + index = dn->node_entry->index; + f2fs_put_cache(dn->node_entry, true); - invalidate_mapping_pages(NODE_MAPPING(sbi), - index, index); + f2fs_invalidate_node_cache(sbi, index); - dn->node_folio = NULL; + dn->node_entry = NULL; trace_f2fs_truncate_node(dn->inode, dn->nid, ni.blk_addr); return 0; @@ -1006,35 +1000,35 @@ static int truncate_node(struct dnode_of_data *dn) static int truncate_dnode(struct dnode_of_data *dn) { struct f2fs_sb_info *sbi = F2FS_I_SB(dn->inode); - struct folio *folio; + struct f2fs_cached_block *entry; int err; if (dn->nid == 0) return 1; /* get direct node */ - folio = f2fs_get_node_folio(sbi, dn->nid, NODE_TYPE_NON_INODE); - if (PTR_ERR(folio) == -ENOENT) + entry = f2fs_get_node_cache(sbi, dn->nid, NODE_TYPE_NON_INODE); + if (PTR_ERR(entry) == -ENOENT) return 1; - else if (IS_ERR(folio)) - return PTR_ERR(folio); + else if (IS_ERR(entry)) + return PTR_ERR(entry); - if (IS_INODE(folio) || ino_of_node(folio) != dn->inode->i_ino) { + if (IS_INODE(entry) || ino_of_node(entry) != dn->inode->i_ino) { f2fs_err(sbi, "incorrect node reference, ino: %llu, nid: %u, ino_of_node: %u", - dn->inode->i_ino, dn->nid, ino_of_node(folio)); + dn->inode->i_ino, dn->nid, ino_of_node(entry)); set_sbi_flag(sbi, SBI_NEED_FSCK); f2fs_handle_error(sbi, ERROR_INVALID_NODE_REFERENCE); - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); return -EFSCORRUPTED; } /* Make dnode_of_data for parameter */ - dn->node_folio = folio; + dn->node_entry = entry; dn->ofs_in_node = 0; f2fs_truncate_data_blocks_range(dn, ADDRS_PER_BLOCK(dn->inode)); err = truncate_node(dn); if (err) { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); return err; } @@ -1045,7 +1039,7 @@ static int truncate_nodes(struct dnode_of_data *dn, unsigned int nofs, int ofs, int depth) { struct dnode_of_data rdn = *dn; - struct folio *folio; + struct f2fs_cached_block *entry; struct f2fs_node *rn; nid_t child_nid; unsigned int child_nofs; @@ -1057,16 +1051,16 @@ static int truncate_nodes(struct dnode_of_data *dn, unsigned int nofs, trace_f2fs_truncate_nodes_enter(dn->inode, dn->nid, dn->data_blkaddr); - folio = f2fs_get_node_folio(F2FS_I_SB(dn->inode), dn->nid, + entry = f2fs_get_node_cache(F2FS_I_SB(dn->inode), dn->nid, NODE_TYPE_NON_INODE); - if (IS_ERR(folio)) { - trace_f2fs_truncate_nodes_exit(dn->inode, PTR_ERR(folio)); - return PTR_ERR(folio); + if (IS_ERR(entry)) { + trace_f2fs_truncate_nodes_exit(dn->inode, PTR_ERR(entry)); + return PTR_ERR(entry); } - f2fs_ra_node_pages(folio, ofs, NIDS_PER_BLOCK); + f2fs_ra_node_pages(entry, ofs, NIDS_PER_BLOCK); - rn = F2FS_NODE(folio); + rn = CACHED_NODE(entry); if (depth < 3) { for (i = ofs; i < NIDS_PER_BLOCK; i++, freed++) { child_nid = le32_to_cpu(rn->in.nid[i]); @@ -1076,7 +1070,7 @@ static int truncate_nodes(struct dnode_of_data *dn, unsigned int nofs, ret = truncate_dnode(&rdn); if (ret < 0) goto out_err; - if (set_nid(folio, i, 0, false)) + if (set_nid(entry, i, 0, false)) dn->node_changed = true; } } else { @@ -1090,7 +1084,7 @@ static int truncate_nodes(struct dnode_of_data *dn, unsigned int nofs, rdn.nid = child_nid; ret = truncate_nodes(&rdn, child_nofs, 0, depth - 1); if (ret == (NIDS_PER_BLOCK + 1)) { - if (set_nid(folio, i, 0, false)) + if (set_nid(entry, i, 0, false)) dn->node_changed = true; child_nofs += ret; } else if (ret < 0 && ret != -ENOENT) { @@ -1102,19 +1096,19 @@ static int truncate_nodes(struct dnode_of_data *dn, unsigned int nofs, if (!ofs) { /* remove current indirect node */ - dn->node_folio = folio; + dn->node_entry = entry; ret = truncate_node(dn); if (ret) goto out_err; freed++; } else { - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); } trace_f2fs_truncate_nodes_exit(dn->inode, freed); return freed; out_err: - f2fs_folio_put(folio, true); + f2fs_put_cache(entry, true); trace_f2fs_truncate_nodes_exit(dn->inode, ret); return ret; } @@ -1122,60 +1116,60 @@ static int truncate_nodes(struct dnode_of_data *dn, unsigned int nofs, static int truncate_partial_nodes(struct dnode_of_data *dn, int *offset, int depth) { - struct folio *folios[2]; + struct f2fs_cached_block *entries[2]; nid_t nid[3]; nid_t child_nid; int err = 0; int i; int idx = depth - 2; - nid[0] = get_nid(dn->inode_folio, offset[0], true); + nid[0] = get_nid(dn->inode_entry, offset[0], true); if (!nid[0]) return 0; /* get indirect nodes in the path */ for (i = 0; i < idx + 1; i++) { /* reference count'll be increased */ - folios[i] = f2fs_get_node_folio(F2FS_I_SB(dn->inode), nid[i], + entries[i] = f2fs_get_node_cache(F2FS_I_SB(dn->inode), nid[i], NODE_TYPE_NON_INODE); - if (IS_ERR(folios[i])) { - err = PTR_ERR(folios[i]); + if (IS_ERR(entries[i])) { + err = PTR_ERR(entries[i]); idx = i - 1; goto fail; } - nid[i + 1] = get_nid(folios[i], offset[i + 1], false); + nid[i + 1] = get_nid(entries[i], offset[i + 1], false); } - f2fs_ra_node_pages(folios[idx], offset[idx + 1], NIDS_PER_BLOCK); + f2fs_ra_node_pages(entries[idx], offset[idx + 1], NIDS_PER_BLOCK); /* free direct nodes linked to a partial indirect node */ for (i = offset[idx + 1]; i < NIDS_PER_BLOCK; i++) { - child_nid = get_nid(folios[idx], i, false); + child_nid = get_nid(entries[idx], i, false); if (!child_nid) continue; dn->nid = child_nid; err = truncate_dnode(dn); if (err < 0) goto fail; - if (set_nid(folios[idx], i, 0, false)) + if (set_nid(entries[idx], i, 0, false)) dn->node_changed = true; } if (offset[idx + 1] == 0) { - dn->node_folio = folios[idx]; + dn->node_entry = entries[idx]; dn->nid = nid[idx]; err = truncate_node(dn); if (err) goto fail; } else { - f2fs_folio_put(folios[idx], true); + f2fs_put_cache(entries[idx], true); } offset[idx]++; offset[idx + 1] = 0; idx--; fail: for (i = idx; i >= 0; i--) - f2fs_folio_put(folios[i], true); + f2fs_put_cache(entries[i], true); trace_f2fs_truncate_partial_nodes(dn->inode, nid, depth, err); @@ -1192,7 +1186,7 @@ int f2fs_truncate_inode_blocks(struct inode *inode, pgoff_t from) int level, offset[4], noffset[4]; unsigned int nofs = 0; struct dnode_of_data dn; - struct folio *folio; + struct f2fs_cached_block *entry; trace_f2fs_truncate_inode_blocks_enter(inode, from); @@ -1209,14 +1203,14 @@ int f2fs_truncate_inode_blocks(struct inode *inode, pgoff_t from) return level; } - folio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(folio)) { - trace_f2fs_truncate_inode_blocks_exit(inode, PTR_ERR(folio)); - return PTR_ERR(folio); + entry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(entry)) { + trace_f2fs_truncate_inode_blocks_exit(inode, PTR_ERR(entry)); + return PTR_ERR(entry); } - set_new_dnode(&dn, inode, folio, NULL, 0); - folio_unlock(folio); + set_new_dnode(&dn, inode, entry, NULL, 0); + f2fs_unlock_cache(entry); switch (level) { case 0: @@ -1246,7 +1240,7 @@ int f2fs_truncate_inode_blocks(struct inode *inode, pgoff_t from) skip_partial: while (cont) { - dn.nid = get_nid(folio, offset[0], true); + dn.nid = get_nid(entry, offset[0], true); switch (offset[0]) { case NODE_DIR1_BLOCK: case NODE_DIR2_BLOCK: @@ -1267,7 +1261,7 @@ int f2fs_truncate_inode_blocks(struct inode *inode, pgoff_t from) BUG(); } if (err == -ENOENT) { - set_sbi_flag(F2FS_F_SB(folio), SBI_NEED_FSCK); + set_sbi_flag(sbi, SBI_NEED_FSCK); f2fs_handle_error(sbi, ERROR_INVALID_BLKADDR); fserror_report_file_metadata(dn.inode, -EFSCORRUPTED, GFP_NOFS); @@ -1280,18 +1274,18 @@ int f2fs_truncate_inode_blocks(struct inode *inode, pgoff_t from) } if (err < 0) goto fail; - if (offset[1] == 0 && get_nid(folio, offset[0], true)) { - folio_lock(folio); - BUG_ON(!is_node_folio(folio)); - set_nid(folio, offset[0], 0, true); - folio_unlock(folio); + if (offset[1] == 0 && get_nid(entry, offset[0], true)) { + f2fs_lock_cache(entry); + f2fs_bug_on(sbi, !f2fs_is_node_cache(entry)); + set_nid(entry, offset[0], 0, true); + f2fs_unlock_cache(entry); } offset[1] = 0; offset[0]++; nofs += err; } fail: - f2fs_folio_put(folio, false); + f2fs_put_cache(entry, false); trace_f2fs_truncate_inode_blocks_exit(inode, err); return err > 0 ? 0 : err; } @@ -1302,20 +1296,20 @@ int f2fs_truncate_xattr_node(struct inode *inode) struct f2fs_sb_info *sbi = F2FS_I_SB(inode); nid_t nid = F2FS_I(inode)->i_xattr_nid; struct dnode_of_data dn; - struct folio *nfolio; + struct f2fs_cached_block *nentry; int err; if (!nid) return 0; - nfolio = f2fs_get_xnode_folio(sbi, nid); - if (IS_ERR(nfolio)) - return PTR_ERR(nfolio); + nentry = f2fs_get_xnode_cache(sbi, nid); + if (IS_ERR(nentry)) + return PTR_ERR(nentry); - set_new_dnode(&dn, inode, NULL, nfolio, nid); + set_new_dnode(&dn, inode, NULL, nentry, nid); err = truncate_node(&dn); if (err) { - f2fs_folio_put(nfolio, true); + f2fs_put_cache(nentry, true); return err; } @@ -1372,30 +1366,30 @@ int f2fs_remove_inode_page(struct inode *inode) return 0; } -struct folio *f2fs_new_inode_folio(struct inode *inode) +struct f2fs_cached_block *f2fs_new_inode_cache(struct inode *inode) { struct dnode_of_data dn; /* allocate inode page for new inode */ set_new_dnode(&dn, inode, NULL, NULL, inode->i_ino); - /* caller should f2fs_folio_put(folio, true); */ - return f2fs_new_node_folio(&dn, 0); + /* caller should f2fs_put_cache(entry, true); */ + return f2fs_new_node_cache(&dn, 0); } -struct folio *f2fs_new_node_folio(struct dnode_of_data *dn, unsigned int ofs) +struct f2fs_cached_block *f2fs_new_node_cache(struct dnode_of_data *dn, unsigned int ofs) { struct f2fs_sb_info *sbi = F2FS_I_SB(dn->inode); struct node_info new_ni; - struct folio *folio; + struct f2fs_cached_block *entry; int err; if (unlikely(is_inode_flag_set(dn->inode, FI_NO_ALLOC))) return ERR_PTR(-EPERM); - folio = f2fs_grab_cache_folio(NODE_MAPPING(sbi), dn->nid, false); - if (IS_ERR(folio)) - return folio; + entry = f2fs_grab_node_cache(sbi, dn->nid); + if (IS_ERR(entry)) + return entry; if (unlikely((err = inc_valid_node_count(sbi, dn->inode, !ofs)))) goto fail; @@ -1411,7 +1405,7 @@ struct folio *f2fs_new_node_folio(struct dnode_of_data *dn, unsigned int ofs) dec_valid_node_count(sbi, dn->inode, !ofs); set_sbi_flag(sbi, SBI_NEED_FSCK); f2fs_warn_ratelimited(sbi, - "f2fs_new_node_folio: inconsistent nat entry, " + "f2fs_new_node_cache: inconsistent nat entry, " "ino:%u, nid:%u, blkaddr:%u, ver:%u, flag:%u", new_ni.ino, new_ni.nid, new_ni.blk_addr, new_ni.version, new_ni.flag); @@ -1426,12 +1420,11 @@ struct folio *f2fs_new_node_folio(struct dnode_of_data *dn, unsigned int ofs) new_ni.version = 0; set_node_addr(sbi, &new_ni, NEW_ADDR, false); - f2fs_folio_wait_writeback(folio, NODE, true, true); - fill_node_footer(folio, dn->nid, dn->inode->i_ino, ofs, true); - set_cold_node(folio, S_ISDIR(dn->inode->i_mode)); - if (!folio_test_uptodate(folio)) - folio_mark_uptodate(folio); - if (folio_mark_dirty(folio)) + f2fs_cache_wait_writeback(entry); + fill_node_footer(entry, dn->nid, dn->inode->i_ino, ofs, true); + set_cold_node(entry, S_ISDIR(dn->inode->i_mode)); + f2fs_cache_set_uptodate(entry); + if (f2fs_mark_cache_dirty(entry)) dn->node_changed = true; if (f2fs_has_xattr_block(ofs)) @@ -1439,54 +1432,54 @@ struct folio *f2fs_new_node_folio(struct dnode_of_data *dn, unsigned int ofs) if (ofs == 0) inc_valid_inode_count(sbi); - return folio; + return entry; fail: - clear_node_folio_dirty(folio); - f2fs_folio_put(folio, true); + f2fs_drop_cache_dirty(entry); + f2fs_put_cache(entry, true); return ERR_PTR(err); } /* * Caller should do after getting the following values. - * 0: f2fs_folio_put(folio, false) - * LOCKED_PAGE or error: f2fs_folio_put(folio, true) + * 0: f2fs_put_cache(cache, false) + * LOCKED_PAGE or error: f2fs_put_cache(entry, true) */ -static int read_node_folio(struct folio *folio, blk_opf_t op_flags) +static int read_node_cache(struct f2fs_cached_block *entry, blk_opf_t op_flags) { - struct f2fs_sb_info *sbi = F2FS_F_SB(folio); + struct f2fs_sb_info *sbi = entry->cache->sbi; struct node_info ni; struct f2fs_io_info fio = { .sbi = sbi, .type = NODE, .op = REQ_OP_READ, .op_flags = op_flags, - .folio = folio, .encrypted_page = NULL, + .cache_entry = entry, + .is_cache = 1, }; int err; - if (folio_test_uptodate(folio)) { - if (!f2fs_inode_chksum_verify(sbi, folio)) { - folio_clear_uptodate(folio); + if (f2fs_cache_test_uptodate(entry)) { + if (!f2fs_inode_chksum_verify(sbi, entry)) { + f2fs_cache_clear_uptodate(entry); return -EFSBADCRC; } return LOCKED_PAGE; } - err = f2fs_get_node_info(sbi, folio->index, &ni, false); + err = f2fs_get_node_info(sbi, entry->index, &ni, false); if (err) return err; /* NEW_ADDR can be seen, after cp_error drops some dirty node pages */ if (unlikely(ni.blk_addr == NULL_ADDR || ni.blk_addr == NEW_ADDR)) { - folio_clear_uptodate(folio); + f2fs_cache_clear_uptodate(entry); return -ENOENT; } fio.new_blkaddr = fio.old_blkaddr = ni.blk_addr; - err = f2fs_submit_page_bio(&fio); - + err = f2fs_submit_cache_read(&fio); if (!err) f2fs_update_iostat(sbi, NULL, FS_NODE_READ_IO, F2FS_BLKSIZE); @@ -1498,7 +1491,7 @@ static int read_node_folio(struct folio *folio, blk_opf_t op_flags) */ void f2fs_ra_node_page(struct f2fs_sb_info *sbi, nid_t nid) { - struct folio *afolio; + struct f2fs_cached_block *entry; int err; if (!nid) @@ -1506,29 +1499,29 @@ void f2fs_ra_node_page(struct f2fs_sb_info *sbi, nid_t nid) if (f2fs_check_nid_range(sbi, nid)) return; - afolio = xa_load(&NODE_MAPPING(sbi)->i_pages, nid); - if (afolio) + entry = xa_load(&META_CACHE(sbi)->root, nid); + if (entry) return; - afolio = f2fs_grab_cache_folio(NODE_MAPPING(sbi), nid, false); - if (IS_ERR(afolio)) + entry = f2fs_grab_node_cache(sbi, nid); + if (IS_ERR(entry)) return; - err = read_node_folio(afolio, REQ_RAHEAD); - f2fs_folio_put(afolio, err ? true : false); + err = read_node_cache(entry, REQ_RAHEAD); + f2fs_put_cache(entry, err ? true : false); } int f2fs_sanity_check_node_footer(struct f2fs_sb_info *sbi, - struct folio *folio, pgoff_t nid, + struct f2fs_cached_block *entry, pgoff_t nid, enum node_type ntype, bool in_irq) { bool is_inode, is_xnode; - if (unlikely(nid != nid_of_node(folio))) + if (unlikely(nid != nid_of_node(entry))) goto out_err; - is_inode = IS_INODE(folio); - is_xnode = f2fs_has_xattr_block(ofs_of_node(folio)); + is_inode = IS_INODE(entry); + is_xnode = f2fs_has_xattr_block(ofs_of_node(entry)); switch (ntype) { case NODE_TYPE_REGULAR: @@ -1561,20 +1554,20 @@ int f2fs_sanity_check_node_footer(struct f2fs_sb_info *sbi, set_sbi_flag(sbi, SBI_NEED_FSCK); f2fs_warn_ratelimited(sbi, "inconsistent node block, node_type:%d, nid:%lu, " "node_footer[nid:%u,ino:%u,ofs:%u,cpver:%llu,blkaddr:%u]", - ntype, nid, nid_of_node(folio), ino_of_node(folio), - ofs_of_node(folio), cpver_of_node(folio), - next_blkaddr_of_node(folio)); + ntype, nid, nid_of_node(entry), ino_of_node(entry), + ofs_of_node(entry), cpver_of_node(entry), + next_blkaddr_of_node(entry)); f2fs_handle_error(sbi, ERROR_INCONSISTENT_FOOTER); - fserror_report_file_metadata(folio->mapping->host, - -EFSCORRUPTED, in_irq ? GFP_NOWAIT : GFP_NOFS); + fserror_report_metadata(sbi->sb, -EFSCORRUPTED, + in_irq ? GFP_NOWAIT : GFP_NOFS); return -EFSCORRUPTED; } -static struct folio *__get_node_folio(struct f2fs_sb_info *sbi, pgoff_t nid, - struct folio *parent, int start, enum node_type ntype) +static struct f2fs_cached_block *__get_node_cache(struct f2fs_sb_info *sbi, pgoff_t nid, + struct f2fs_cached_block *parent, int start, enum node_type ntype) { - struct folio *folio; + struct f2fs_cached_block *entry; int err; if (!nid) @@ -1582,71 +1575,71 @@ static struct folio *__get_node_folio(struct f2fs_sb_info *sbi, pgoff_t nid, if (f2fs_check_nid_range(sbi, nid)) return ERR_PTR(-EINVAL); repeat: - folio = f2fs_grab_cache_folio(NODE_MAPPING(sbi), nid, false); - if (IS_ERR(folio)) - return folio; + entry = f2fs_grab_node_cache(sbi, nid); + if (IS_ERR(entry)) + return entry; - err = read_node_folio(folio, 0); + err = read_node_cache(entry, 0); if (err < 0) goto out_put_err; if (err == LOCKED_PAGE) - goto page_hit; + goto entry_hit; if (parent) f2fs_ra_node_pages(parent, start + 1, MAX_RA_NODE); - folio_lock(folio); + f2fs_lock_cache(entry); - if (unlikely(!is_node_folio(folio))) { - f2fs_folio_put(folio, true); + if (unlikely(!f2fs_is_node_cache(entry))) { + f2fs_put_cache(entry, true); goto repeat; } - if (unlikely(!folio_test_uptodate(folio))) { + if (unlikely(!f2fs_cache_test_uptodate(entry))) { err = -EIO; goto out_put_err; } - if (!f2fs_inode_chksum_verify(sbi, folio)) { + if (!f2fs_inode_chksum_verify(sbi, entry)) { err = -EFSBADCRC; goto out_err; } -page_hit: - err = f2fs_sanity_check_node_footer(sbi, folio, nid, ntype, false); +entry_hit: + err = f2fs_sanity_check_node_footer(sbi, entry, nid, ntype, false); if (!err) - return folio; + return entry; out_err: - clear_node_folio_dirty(folio); + f2fs_clear_cache_dirty(entry); out_put_err: /* ENOENT comes from read_node_folio which is not an error. */ if (err != -ENOENT) - f2fs_handle_page_eio(sbi, folio->index, NODE); - f2fs_folio_put(folio, true); + f2fs_handle_page_eio(sbi, entry->index, NODE); + f2fs_put_cache(entry, true); return ERR_PTR(err); } -struct folio *f2fs_get_node_folio(struct f2fs_sb_info *sbi, pgoff_t nid, +struct f2fs_cached_block *f2fs_get_node_cache(struct f2fs_sb_info *sbi, pgoff_t nid, enum node_type node_type) { - return __get_node_folio(sbi, nid, NULL, 0, node_type); + return __get_node_cache(sbi, nid, NULL, 0, node_type); } -struct folio *f2fs_get_inode_folio(struct f2fs_sb_info *sbi, pgoff_t ino) +struct f2fs_cached_block *f2fs_get_inode_cache(struct f2fs_sb_info *sbi, pgoff_t ino) { - return __get_node_folio(sbi, ino, NULL, 0, NODE_TYPE_INODE); + return __get_node_cache(sbi, ino, NULL, 0, NODE_TYPE_INODE); } -struct folio *f2fs_get_xnode_folio(struct f2fs_sb_info *sbi, pgoff_t xnid) +struct f2fs_cached_block *f2fs_get_xnode_cache(struct f2fs_sb_info *sbi, pgoff_t xnid) { - return __get_node_folio(sbi, xnid, NULL, 0, NODE_TYPE_XATTR); + return __get_node_cache(sbi, xnid, NULL, 0, NODE_TYPE_XATTR); } -static struct folio *f2fs_get_node_folio_ra(struct folio *parent, int start) +static struct f2fs_cached_block *f2fs_get_node_cache_ra(struct f2fs_cached_block *parent, int start) { - struct f2fs_sb_info *sbi = F2FS_F_SB(parent); + struct f2fs_sb_info *sbi = parent->cache->sbi; nid_t nid = get_nid(parent, start, false); - return __get_node_folio(sbi, nid, parent, start, NODE_TYPE_NON_IXNODE); + return __get_node_cache(sbi, nid, parent, start, NODE_TYPE_NON_IXNODE); } static void flush_inline_data(struct f2fs_sb_info *sbi, nid_t ino) @@ -1685,110 +1678,103 @@ static void flush_inline_data(struct f2fs_sb_info *sbi, nid_t ino) iput(inode); } -static struct folio *last_fsync_dnode(struct f2fs_sb_info *sbi, nid_t ino) +static struct f2fs_cached_block *last_fsync_dnode(struct f2fs_sb_info *sbi, nid_t ino) { - pgoff_t index; - struct folio_batch fbatch; - struct folio *last_folio = NULL; - int nr_folios; - - folio_batch_init(&fbatch); - index = 0; + pgoff_t index = 0; + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; + struct f2fs_cached_block *last_entry = NULL; + unsigned int nr; - while ((nr_folios = filemap_get_folios_tag(NODE_MAPPING(sbi), &index, - (pgoff_t)-1, PAGECACHE_TAG_DIRTY, - &fbatch))) { + while ((nr = f2fs_cache_gang_lookup_tag(NODE_CACHE(sbi), + entries, &index, F2FS_ONSTACK_CACHES, + F2FS_CACHE_TAG_DIRTY))) { int i; - for (i = 0; i < nr_folios; i++) { - struct folio *folio = fbatch.folios[i]; + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; if (unlikely(f2fs_cp_error(sbi))) { - f2fs_folio_put(last_folio, false); - folio_batch_release(&fbatch); + f2fs_put_cache(last_entry, false); + f2fs_cache_gang_release(entries, nr); return ERR_PTR(-EIO); } - if (!IS_DNODE(folio) || !is_cold_node(folio)) + if (!IS_DNODE(entry) || !is_cold_node(entry)) continue; - if (ino_of_node(folio) != ino) + if (ino_of_node(entry) != ino) continue; - folio_lock(folio); + f2fs_lock_cache(entry); - if (unlikely(!is_node_folio(folio))) { + if (unlikely(!f2fs_is_node_cache(entry))) { continue_unlock: - folio_unlock(folio); + f2fs_unlock_cache(entry); continue; } - if (ino_of_node(folio) != ino) + if (ino_of_node(entry) != ino) goto continue_unlock; - if (!folio_test_dirty(folio)) { + if (!f2fs_cache_test_dirty(entry)) { /* someone wrote it for us */ goto continue_unlock; } - if (last_folio) - f2fs_folio_put(last_folio, false); + if (last_entry) + f2fs_put_cache(last_entry, false); - folio_get(folio); - last_folio = folio; - folio_unlock(folio); + f2fs_cache_get(entry); + last_entry = entry; + f2fs_unlock_cache(entry); } - folio_batch_release(&fbatch); + f2fs_cache_gang_release(entries, nr); cond_resched(); } - return last_folio; + return last_entry; } -static bool __write_node_folio(struct folio *folio, bool atomic, bool do_fsync, - bool *submitted, struct writeback_control *wbc, - bool do_balance, enum iostat_type io_type, - unsigned int *seq_id) +static bool __write_node_cache(struct f2fs_cached_block *entry, + bool atomic, bool do_fsync, bool *submitted, + bool sync, bool do_balance, + enum iostat_type io_type, unsigned int *seq_id) { - struct f2fs_sb_info *sbi = F2FS_F_SB(folio); + struct f2fs_sb_info *sbi = entry->cache->sbi; nid_t nid; struct node_info ni; struct f2fs_io_info fio = { .sbi = sbi, - .ino = ino_of_node(folio), + .ino = ino_of_node(entry), .type = NODE, .op = REQ_OP_WRITE, - .op_flags = wbc_to_write_flags(wbc), - .folio = folio, + .op_flags = sync ? REQ_SYNC : REQ_BACKGROUND, + .cache_entry = entry, .encrypted_page = NULL, .submitted = 0, .io_type = io_type, - .io_wbc = wbc, + .is_cache = true, }; struct f2fs_lock_context lc; unsigned int seq; - trace_f2fs_writepage(folio, NODE); - if (unlikely(f2fs_cp_error(sbi))) { /* keep node pages in remount-ro mode */ if (F2FS_OPTION(sbi).errors == MOUNT_ERRORS_READONLY) goto redirty_out; - folio_clear_uptodate(folio); - dec_page_count(sbi, F2FS_DIRTY_NODES); - folio_unlock(folio); + f2fs_force_clear_cache_dirty(entry); + f2fs_unlock_cache(entry); return true; } if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING))) goto redirty_out; - if (!is_sbi_flag_set(sbi, SBI_CP_DISABLED) && - wbc->sync_mode == WB_SYNC_NONE && - IS_DNODE(folio) && is_cold_node(folio)) + if (!is_sbi_flag_set(sbi, SBI_CP_DISABLED) && !sync && + IS_DNODE(entry) && is_cold_node(entry)) goto redirty_out; /* get old block addr of this node page */ - nid = nid_of_node(folio); + nid = nid_of_node(entry); - if (f2fs_sanity_check_node_footer(sbi, folio, folio->index, + if (f2fs_sanity_check_node_footer(sbi, entry, entry->index, NODE_TYPE_REGULAR, false)) { fserror_report_metadata(sbi->sb, -EFSCORRUPTED, GFP_NOFS); f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_CORRUPTED_NID); @@ -1802,10 +1788,10 @@ static bool __write_node_folio(struct folio *folio, bool atomic, bool do_fsync, /* This page is already truncated */ if (unlikely(ni.blk_addr == NULL_ADDR)) { - folio_clear_uptodate(folio); + f2fs_cache_clear_uptodate(entry); dec_page_count(sbi, F2FS_DIRTY_NODES); f2fs_up_read_trace(&sbi->node_write, &lc); - folio_unlock(folio); + f2fs_unlock_cache(entry); return true; } @@ -1819,28 +1805,28 @@ static bool __write_node_folio(struct folio *folio, bool atomic, bool do_fsync, if (atomic && !test_opt(sbi, NOBARRIER)) fio.op_flags |= REQ_PREFLUSH | REQ_FUA; - set_dentry_mark(folio, false); - set_fsync_mark(folio, do_fsync); - if (IS_INODE(folio) && (atomic || is_fsync_dnode(folio))) - set_dentry_mark(folio, - f2fs_need_dentry_mark(sbi, ino_of_node(folio))); + set_dentry_mark(entry, false); + set_fsync_mark(entry, do_fsync); + if (IS_INODE(entry) && (atomic || is_fsync_dnode(entry))) + set_dentry_mark(entry, + f2fs_need_dentry_mark(sbi, ino_of_node(entry))); /* should add to global list before clearing PAGECACHE status */ - if (f2fs_in_warm_node_list(folio)) { - seq = f2fs_add_fsync_node_entry(sbi, folio); + if (f2fs_in_warm_node_list(entry)) { + seq = f2fs_add_fsync_node_entry(sbi, entry); if (seq_id) *seq_id = seq; } - folio_start_writeback(folio); + f2fs_start_cache_writeback(entry); fio.old_blkaddr = ni.blk_addr; f2fs_do_write_node_page(nid, &fio); - set_node_addr(sbi, &ni, fio.new_blkaddr, is_fsync_dnode(folio)); + set_node_addr(sbi, &ni, fio.new_blkaddr, is_fsync_dnode(entry)); dec_page_count(sbi, F2FS_DIRTY_NODES); f2fs_up_read_trace(&sbi->node_write, &lc); - folio_unlock(folio); + f2fs_unlock_cache(entry); if (unlikely(f2fs_cp_error(sbi))) { f2fs_submit_merged_write(sbi, NODE); @@ -1854,171 +1840,166 @@ static bool __write_node_folio(struct folio *folio, bool atomic, bool do_fsync, return true; redirty_out: - folio_redirty_for_writepage(wbc, folio); - folio_unlock(folio); + f2fs_cache_set_dirty(entry); + f2fs_unlock_cache(entry); return false; } -int f2fs_write_single_node_folio(struct folio *node_folio, int sync_mode, +int f2fs_write_node_cache(struct f2fs_cached_block *node_entry, int sync_mode, bool mark_dirty, enum iostat_type io_type) { int err = 0; - struct writeback_control wbc = { - .sync_mode = WB_SYNC_ALL, - .nr_to_write = 1, - }; if (!sync_mode) { /* set page dirty and write it */ - if (!folio_test_writeback(node_folio)) - folio_mark_dirty(node_folio); - goto out_folio; + if (!f2fs_cache_test_writeback(node_entry)) + f2fs_mark_cache_dirty(node_entry); + goto out_entry; } - f2fs_folio_wait_writeback(node_folio, NODE, true, true); + f2fs_cache_wait_writeback(node_entry); if (mark_dirty) - folio_mark_dirty(node_folio); - else if (!folio_test_dirty(node_folio)) - goto out_folio; + f2fs_mark_cache_dirty(node_entry); + else if (!f2fs_cache_test_dirty(node_entry)) + goto out_entry; - if (!folio_clear_dirty_for_io(node_folio)) { + if (!f2fs_clear_cache_dirty(node_entry)) { err = -EAGAIN; - goto out_folio; + goto out_entry; } - if (!__write_node_folio(node_folio, false, false, NULL, - &wbc, false, io_type, NULL)) + if (!__write_node_cache(node_entry, false, false, NULL, + true, false, io_type, NULL)) err = -EAGAIN; - goto release_folio; -out_folio: - folio_unlock(node_folio); -release_folio: - f2fs_folio_put(node_folio, false); + goto release_entry; +out_entry: + f2fs_unlock_cache(node_entry); +release_entry: + f2fs_put_cache(node_entry, false); return err; } -int f2fs_move_node_folio(struct folio *node_folio, int gc_type) +int f2fs_move_node_cache(struct f2fs_cached_block *entry, int gc_type) { - return f2fs_write_single_node_folio(node_folio, gc_type == FG_GC, + return f2fs_write_node_cache(entry, gc_type == FG_GC, true, FS_GC_NODE_IO); } -int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode, - struct writeback_control *wbc, bool atomic, - unsigned int *seq_id) +int f2fs_fsync_node_caches(struct f2fs_sb_info *sbi, struct inode *inode, + bool atomic, unsigned int *seq_id) { pgoff_t index; - struct folio_batch fbatch; + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; int ret = 0; - struct folio *last_folio = NULL; + struct f2fs_cached_block *last_entry = NULL; bool marked = false; nid_t ino = inode->i_ino; - int nr_folios; + int nr; int nwritten = 0; if (atomic) { - last_folio = last_fsync_dnode(sbi, ino); - if (IS_ERR_OR_NULL(last_folio)) - return PTR_ERR_OR_ZERO(last_folio); + last_entry = last_fsync_dnode(sbi, ino); + if (IS_ERR_OR_NULL(last_entry)) + return PTR_ERR_OR_ZERO(last_entry); } retry: - folio_batch_init(&fbatch); index = 0; - while ((nr_folios = filemap_get_folios_tag(NODE_MAPPING(sbi), &index, - (pgoff_t)-1, PAGECACHE_TAG_DIRTY, - &fbatch))) { + while ((nr = f2fs_cache_gang_lookup_tag(NODE_CACHE(sbi), + entries, &index, F2FS_ONSTACK_CACHES, + F2FS_CACHE_TAG_DIRTY))) { int i; - for (i = 0; i < nr_folios; i++) { - struct folio *folio = fbatch.folios[i]; + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; bool submitted = false; bool do_fsync = false; if (unlikely(f2fs_cp_error(sbi))) { - f2fs_folio_put(last_folio, false); - folio_batch_release(&fbatch); + f2fs_put_cache(last_entry, false); + f2fs_cache_gang_release(entries, nr); ret = -EIO; goto out; } - if (!IS_DNODE(folio) || !is_cold_node(folio)) + if (!IS_DNODE(entry) || !is_cold_node(entry)) continue; - if (ino_of_node(folio) != ino) + if (ino_of_node(entry) != ino) continue; - folio_lock(folio); + f2fs_lock_cache(entry); - if (unlikely(!is_node_folio(folio))) { + if (unlikely(!f2fs_is_node_cache(entry))) { continue_unlock: - folio_unlock(folio); + f2fs_unlock_cache(entry); continue; } - if (ino_of_node(folio) != ino) + if (ino_of_node(entry) != ino) goto continue_unlock; - if (!folio_test_dirty(folio) && folio != last_folio) { + if (!f2fs_cache_test_dirty(entry) && entry != last_entry) { /* someone wrote it for us */ goto continue_unlock; } - f2fs_folio_wait_writeback(folio, NODE, true, true); + f2fs_cache_wait_writeback(entry); - if (!atomic || folio == last_folio) { + if (!atomic || entry == last_entry) { do_fsync = true; percpu_counter_inc(&sbi->rf_node_block_count); - if (IS_INODE(folio)) { + if (IS_INODE(entry)) { if (is_inode_flag_set(inode, FI_DIRTY_INODE)) - f2fs_update_inode(inode, folio); + f2fs_update_inode(inode, entry); } /* may be written by other thread */ - if (!folio_test_dirty(folio)) - folio_mark_dirty(folio); + if (!f2fs_cache_test_dirty(entry)) + f2fs_mark_cache_dirty(entry); } - if (!folio_clear_dirty_for_io(folio)) + if (!f2fs_clear_cache_dirty(entry)) goto continue_unlock; - if (!__write_node_folio(folio, atomic && - folio == last_folio, + if (!__write_node_cache(entry, atomic && + entry == last_entry, do_fsync, &submitted, - wbc, true, FS_NODE_IO, + true, true, FS_NODE_IO, seq_id)) { - f2fs_folio_put(last_folio, false); - folio_batch_release(&fbatch); + f2fs_put_cache(last_entry, false); + f2fs_cache_gang_release(entries, nr); ret = -EIO; goto out; } if (submitted) nwritten++; - if (folio == last_folio) { - f2fs_folio_put(folio, false); - folio_batch_release(&fbatch); + if (entry == last_entry) { + f2fs_put_cache(entry, false); + f2fs_cache_gang_release(entries, nr); marked = true; goto out; } } - folio_batch_release(&fbatch); + f2fs_cache_gang_release(entries, nr); cond_resched(); } if (atomic && !marked) { f2fs_debug(sbi, "Retry to write fsync mark: ino=%u, idx=%lx", - ino, last_folio->index); - folio_lock(last_folio); - f2fs_folio_wait_writeback(last_folio, NODE, true, true); - folio_mark_dirty(last_folio); - folio_unlock(last_folio); + ino, last_entry->index); + f2fs_lock_cache(last_entry); + f2fs_cache_wait_writeback(last_entry); + f2fs_mark_cache_dirty(last_entry); + f2fs_unlock_cache(last_entry); goto retry; } out: if (nwritten) - f2fs_submit_merged_write_cond(sbi, NULL, NULL, ino, NODE); + f2fs_submit_merged_write_cache(sbi, NULL, ino, NODE); return ret; } + static int f2fs_match_ino(struct inode *inode, u64 ino, void *data) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); @@ -2043,18 +2024,18 @@ static int f2fs_match_ino(struct inode *inode, u64 ino, void *data) return 1; } -static bool flush_dirty_inode(struct folio *folio) +static bool flush_dirty_cache_inode(struct f2fs_cached_block *entry) { - struct f2fs_sb_info *sbi = F2FS_F_SB(folio); + struct f2fs_sb_info *sbi = entry->cache->sbi; struct inode *inode; - nid_t ino = ino_of_node(folio); + nid_t ino = ino_of_node(entry); inode = find_inode_nowait(sbi->sb, ino, f2fs_match_ino, NULL); if (!inode) return false; - f2fs_update_inode(inode, folio); - folio_unlock(folio); + f2fs_update_inode(inode, entry); + f2fs_unlock_cache(entry); iput(inode); return true; @@ -2063,72 +2044,71 @@ static bool flush_dirty_inode(struct folio *folio) void f2fs_flush_inline_data(struct f2fs_sb_info *sbi) { pgoff_t index = 0; - struct folio_batch fbatch; - int nr_folios; + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; + unsigned int nr; - folio_batch_init(&fbatch); - - while ((nr_folios = filemap_get_folios_tag(NODE_MAPPING(sbi), &index, - (pgoff_t)-1, PAGECACHE_TAG_DIRTY, - &fbatch))) { + while ((nr = f2fs_cache_gang_lookup_tag(NODE_CACHE(sbi), + entries, &index, F2FS_ONSTACK_CACHES, + F2FS_CACHE_TAG_DIRTY))) { int i; - for (i = 0; i < nr_folios; i++) { - struct folio *folio = fbatch.folios[i]; + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; - if (!IS_INODE(folio)) + if (!IS_INODE(entry)) continue; - folio_lock(folio); + f2fs_lock_cache(entry); - if (unlikely(!is_node_folio(folio))) + if (unlikely(!f2fs_is_node_cache(entry))) goto unlock; - if (!folio_test_dirty(folio)) + if (!f2fs_cache_test_dirty(entry)) goto unlock; /* flush inline_data, if it's async context. */ - if (folio_test_f2fs_inline(folio)) { - folio_clear_f2fs_inline(folio); - folio_unlock(folio); - flush_inline_data(sbi, ino_of_node(folio)); + if (f2fs_cache_test_inline(entry)) { + f2fs_cache_clear_inline(entry); + f2fs_unlock_cache(entry); + flush_inline_data(sbi, ino_of_node(entry)); continue; } unlock: - folio_unlock(folio); + f2fs_unlock_cache(entry); } - folio_batch_release(&fbatch); + f2fs_cache_gang_release(entries, nr); cond_resched(); } } -int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, - struct writeback_control *wbc, - bool do_balance, enum iostat_type io_type) +int f2fs_writeback_node_caches(struct f2fs_sb_info *sbi, long nr_to_write, + bool sync, bool do_balance, enum iostat_type io_type) { pgoff_t index; - struct folio_batch fbatch; + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; int step = 0; int nwritten = 0; int ret = 0; - int nr_folios, done = 0; - - folio_batch_init(&fbatch); + int nr, done = 0; next_step: index = 0; - while (!done && (nr_folios = filemap_get_folios_tag(NODE_MAPPING(sbi), - &index, (pgoff_t)-1, PAGECACHE_TAG_DIRTY, - &fbatch))) { + while (!done && (nr = f2fs_cache_gang_lookup_tag(NODE_CACHE(sbi), + entries, &index, F2FS_ONSTACK_CACHES, + F2FS_CACHE_TAG_DIRTY))) { int i; - for (i = 0; i < nr_folios; i++) { - struct folio *folio = fbatch.folios[i]; + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; bool submitted = false; + if (!sync && unlikely(freezing(current))) { + done = 1; + break; + } + /* give a priority to WB_SYNC threads */ - if (atomic_read(&sbi->wb_sync_req[NODE]) && - wbc->sync_mode == WB_SYNC_NONE) { + if (atomic_read(&sbi->wb_sync_req[NODE]) && !sync) { done = 1; break; } @@ -2139,27 +2119,27 @@ int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, * 1. dentry dnodes * 2. file dnodes */ - if (step == 0 && IS_DNODE(folio)) + if (step == 0 && IS_DNODE(entry)) continue; - if (step == 1 && (!IS_DNODE(folio) || - is_cold_node(folio))) + if (step == 1 && (!IS_DNODE(entry) || + is_cold_node(entry))) continue; - if (step == 2 && (!IS_DNODE(folio) || - !is_cold_node(folio))) + if (step == 2 && (!IS_DNODE(entry) || + !is_cold_node(entry))) continue; lock_node: - if (wbc->sync_mode == WB_SYNC_ALL) - folio_lock(folio); - else if (!folio_trylock(folio)) + if (sync) + f2fs_lock_cache(entry); + else if (!f2fs_trylock_cache(entry)) continue; - if (unlikely(!is_node_folio(folio))) { + if (unlikely(!f2fs_is_node_cache(entry))) { continue_unlock: - folio_unlock(folio); + f2fs_unlock_cache(entry); continue; } - if (!folio_test_dirty(folio)) { + if (!f2fs_cache_test_dirty(entry)) { /* someone wrote it for us */ goto continue_unlock; } @@ -2169,38 +2149,38 @@ int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, goto write_node; /* flush inline_data */ - if (folio_test_f2fs_inline(folio)) { - folio_clear_f2fs_inline(folio); - folio_unlock(folio); - flush_inline_data(sbi, ino_of_node(folio)); + if (f2fs_cache_test_inline(entry)) { + f2fs_cache_clear_inline(entry); + f2fs_unlock_cache(entry); + flush_inline_data(sbi, ino_of_node(entry)); goto lock_node; } /* flush dirty inode */ - if (IS_INODE(folio) && flush_dirty_inode(folio)) + if (IS_INODE(entry) && flush_dirty_cache_inode(entry)) goto lock_node; write_node: - f2fs_folio_wait_writeback(folio, NODE, true, true); + f2fs_cache_wait_writeback(entry); - if (!folio_clear_dirty_for_io(folio)) + if (!f2fs_clear_cache_dirty(entry)) goto continue_unlock; - if (!__write_node_folio(folio, false, false, &submitted, - wbc, do_balance, io_type, NULL)) { - folio_batch_release(&fbatch); + if (!__write_node_cache(entry, false, false, &submitted, + sync, do_balance, io_type, NULL)) { + f2fs_cache_gang_release(entries, nr); ret = -EIO; goto out; } if (submitted) nwritten++; - if (--wbc->nr_to_write == 0) + if (--nr_to_write == 0) break; } - folio_batch_release(&fbatch); + f2fs_cache_gang_release(entries, nr); cond_resched(); - if (wbc->nr_to_write == 0) { + if (nr_to_write == 0) { step = 2; break; } @@ -2208,7 +2188,7 @@ int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, if (step < 2) { if (!is_sbi_flag_set(sbi, SBI_CP_DISABLED) && - wbc->sync_mode == WB_SYNC_NONE && step == 1) + !sync && step == 1) goto out; step++; goto next_step; @@ -2222,7 +2202,7 @@ int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, return ret; } -int f2fs_wait_on_node_pages_writeback(struct f2fs_sb_info *sbi, +int f2fs_wait_on_node_caches_writeback(struct f2fs_sb_info *sbi, unsigned int seq_id) { struct fsync_node_entry *fn; @@ -2231,7 +2211,7 @@ int f2fs_wait_on_node_pages_writeback(struct f2fs_sb_info *sbi, unsigned int cur_seq_id = 0; while (seq_id && cur_seq_id < seq_id) { - struct folio *folio; + struct f2fs_cached_block *entry; spin_lock_irqsave(&sbi->fsync_node_lock, flags); if (list_empty(head)) { @@ -2244,94 +2224,48 @@ int f2fs_wait_on_node_pages_writeback(struct f2fs_sb_info *sbi, break; } cur_seq_id = fn->seq_id; - folio = fn->folio; - folio_get(folio); + entry = fn->entry; + f2fs_cache_get(entry); spin_unlock_irqrestore(&sbi->fsync_node_lock, flags); - f2fs_folio_wait_writeback(folio, NODE, true, false); + f2fs_cache_wait_writeback(entry); - folio_put(folio); + f2fs_put_cache(entry, false); } - return filemap_check_errors(NODE_MAPPING(sbi)); + return f2fs_cp_error(sbi) ? -EIO : 0; } -static int f2fs_write_node_pages(struct address_space *mapping, - struct writeback_control *wbc) +int f2fs_write_node_caches(struct f2fs_sb_info *sbi) { - struct f2fs_sb_info *sbi = F2FS_M_SB(mapping); struct blk_plug plug; - long diff; + long nr_to_write = LONG_MAX; if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING))) - goto skip_write; + return -EAGAIN; /* balancing f2fs's metadata in background */ f2fs_balance_fs_bg(sbi, true); /* collect a number of dirty node pages and write together */ - if (wbc->sync_mode != WB_SYNC_ALL && - get_pages(sbi, F2FS_DIRTY_NODES) < + if (get_pages(sbi, F2FS_DIRTY_NODES) < nr_pages_to_skip(sbi, NODE)) - goto skip_write; + return -EAGAIN; - if (wbc->sync_mode == WB_SYNC_ALL) - atomic_inc(&sbi->wb_sync_req[NODE]); - else if (atomic_read(&sbi->wb_sync_req[NODE])) { + if (atomic_read(&sbi->wb_sync_req[NODE])) { /* to avoid potential deadlock */ if (current->plug) blk_finish_plug(current->plug); - goto skip_write; + return -EAGAIN; } - trace_f2fs_writepages(mapping->host, wbc, NODE); - - diff = nr_pages_to_write(sbi, NODE, wbc); + nr_to_write = adjust_flush_cache_number(sbi, NODE); blk_start_plug(&plug); - f2fs_sync_node_pages(sbi, wbc, true, FS_NODE_IO); + f2fs_writeback_node_caches(sbi, nr_to_write, false, true, FS_NODE_IO); blk_finish_plug(&plug); - wbc->nr_to_write = max((long)0, wbc->nr_to_write - diff); - - if (wbc->sync_mode == WB_SYNC_ALL) - atomic_dec(&sbi->wb_sync_req[NODE]); - return 0; - -skip_write: - wbc->pages_skipped += get_pages(sbi, F2FS_DIRTY_NODES); - trace_f2fs_writepages(mapping->host, wbc, NODE); return 0; } -static bool f2fs_dirty_node_folio(struct address_space *mapping, - struct folio *folio) -{ - trace_f2fs_set_page_dirty(folio, NODE); - - if (!folio_test_uptodate(folio)) - folio_mark_uptodate(folio); -#ifdef CONFIG_F2FS_CHECK_FS - if (IS_INODE(folio)) - f2fs_inode_chksum_set(F2FS_M_SB(mapping), folio); -#endif - if (filemap_dirty_folio(mapping, folio)) { - inc_page_count(F2FS_M_SB(mapping), F2FS_DIRTY_NODES); - folio_set_f2fs_reference(folio); - return true; - } - return false; -} - -/* - * Structure of the f2fs node operations - */ -const struct address_space_operations f2fs_node_aops = { - .writepages = f2fs_write_node_pages, - .dirty_folio = f2fs_dirty_node_folio, - .invalidate_folio = f2fs_invalidate_folio, - .release_folio = f2fs_release_folio, - .migrate_folio = filemap_migrate_folio, -}; - static struct free_nid *__lookup_free_nid_list(struct f2fs_nm_info *nm_i, nid_t n) { @@ -2829,12 +2763,12 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct f2fs_cached_block *ent { void *src_addr, *dst_addr; size_t inline_size; - struct folio *ifolio; + struct f2fs_cached_block *ientry; struct f2fs_inode *ri; - ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino); - if (IS_ERR(ifolio)) - return PTR_ERR(ifolio); + ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino); + if (IS_ERR(ientry)) + return PTR_ERR(ientry); ri = &CACHED_NODE(entry)->i; if (ri->i_inline & F2FS_INLINE_XATTR) { @@ -2850,15 +2784,15 @@ int f2fs_recover_inline_xattr(struct inode *inode, struct f2fs_cached_block *ent goto update_inode; } - dst_addr = inline_xattr_addr(inode, ifolio); - src_addr = inline_xattr_addr(inode, cache_folio(entry)); + dst_addr = inline_xattr_addr(inode, ientry); + src_addr = inline_xattr_addr(inode, entry); inline_size = inline_xattr_size(inode); - f2fs_folio_wait_writeback(ifolio, NODE, true, true); + f2fs_cache_wait_writeback(ientry); memcpy(dst_addr, src_addr, inline_size); update_inode: - f2fs_update_inode(inode, ifolio); - f2fs_folio_put(ifolio, true); + f2fs_update_inode(inode, ientry); + f2fs_put_cache(ientry, true); return 0; } @@ -2869,7 +2803,7 @@ int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry nid_t new_xnid; struct dnode_of_data dn; struct node_info ni; - struct folio *xfolio; + struct f2fs_cached_block *xentry; int err; if (!prev_xnid) @@ -2890,10 +2824,10 @@ int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry return -ENOSPC; set_new_dnode(&dn, inode, NULL, NULL, new_xnid); - xfolio = f2fs_new_node_folio(&dn, XATTR_NODE_OFFSET); - if (IS_ERR(xfolio)) { + xentry = f2fs_new_node_cache(&dn, XATTR_NODE_OFFSET); + if (IS_ERR(xentry)) { f2fs_alloc_nid_failed(sbi, new_xnid); - return PTR_ERR(xfolio); + return PTR_ERR(xentry); } f2fs_alloc_nid_done(sbi, new_xnid); @@ -2901,11 +2835,11 @@ int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry /* 3: update and set xattr node page dirty */ if (entry) { - memcpy(F2FS_NODE(xfolio), CACHED_NODE(entry), + memcpy(CACHED_NODE(xentry), CACHED_NODE(entry), VALID_XATTR_BLOCK_SIZE); - folio_mark_dirty(xfolio); + f2fs_mark_cache_dirty(xentry); } - f2fs_folio_put(xfolio, true); + f2fs_put_cache(xentry, true); return 0; } @@ -2913,9 +2847,9 @@ int f2fs_recover_xattr_data(struct inode *inode, struct f2fs_cached_block *entry int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry) { struct f2fs_inode *src, *dst; - nid_t ino = ino_of_node(cache_folio(entry)); + nid_t ino = ino_of_node(entry); struct node_info old_ni, new_ni; - struct folio *ifolio; + struct f2fs_cached_block *ientry; int err; err = f2fs_get_node_info(sbi, ino, &old_ni, false); @@ -2925,8 +2859,8 @@ int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block * if (unlikely(old_ni.blk_addr != NULL_ADDR)) return -EINVAL; retry: - ifolio = f2fs_grab_cache_folio(NODE_MAPPING(sbi), ino, false); - if (IS_ERR(ifolio)) { + ientry = f2fs_grab_node_cache(sbi, ino); + if (IS_ERR(ientry)) { memalloc_retry_wait(GFP_NOFS); goto retry; } @@ -2934,13 +2868,12 @@ int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block * /* Should not use this inode from free nid list */ remove_free_nid(sbi, ino); - if (!folio_test_uptodate(ifolio)) - folio_mark_uptodate(ifolio); - fill_node_footer(ifolio, ino, ino, 0, true); - set_cold_node(ifolio, false); + f2fs_cache_set_uptodate(ientry); + fill_node_footer(ientry, ino, ino, 0, true); + set_cold_node(ientry, false); src = &CACHED_NODE(entry)->i; - dst = F2FS_INODE(ifolio); + dst = F2FS_INODE(ientry); memcpy(dst, src, offsetof(struct f2fs_inode, i_ext)); dst->i_size = 0; @@ -2976,8 +2909,8 @@ int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct f2fs_cached_block * WARN_ON(1); set_node_addr(sbi, &new_ni, NEW_ADDR, false); inc_valid_inode_count(sbi); - folio_mark_dirty(ifolio); - f2fs_folio_put(ifolio, true); + f2fs_mark_cache_dirty(ientry); + f2fs_put_cache(ientry, true); return 0; } diff --git a/fs/f2fs/node.h b/fs/f2fs/node.h index de8dcd5d4392..e37a0cb708f9 100644 --- a/fs/f2fs/node.h +++ b/fs/f2fs/node.h @@ -240,41 +240,36 @@ static inline void set_to_next_nat(struct f2fs_nm_info *nm_i, nid_t start_nid) #endif } -static inline nid_t ino_of_node(const struct folio *node_folio) +static inline nid_t ino_of_node(const struct f2fs_cached_block *entry) { - struct f2fs_node *rn = F2FS_NODE(node_folio); - return le32_to_cpu(rn->footer.ino); + return le32_to_cpu(CACHED_NODE(entry)->footer.ino); } -static inline nid_t nid_of_node(const struct folio *node_folio) +static inline nid_t nid_of_node(const struct f2fs_cached_block *entry) { - struct f2fs_node *rn = F2FS_NODE(node_folio); - return le32_to_cpu(rn->footer.nid); + return le32_to_cpu(CACHED_NODE(entry)->footer.nid); } -static inline unsigned int ofs_of_node(const struct folio *node_folio) +static inline unsigned int ofs_of_node(const struct f2fs_cached_block *entry) { - struct f2fs_node *rn = F2FS_NODE(node_folio); - unsigned flag = le32_to_cpu(rn->footer.flag); + unsigned int flag = le32_to_cpu(CACHED_NODE(entry)->footer.flag); return flag >> OFFSET_BIT_SHIFT; } -static inline __u64 cpver_of_node(const struct folio *node_folio) +static inline __u64 cpver_of_node(const struct f2fs_cached_block *entry) { - struct f2fs_node *rn = F2FS_NODE(node_folio); - return le64_to_cpu(rn->footer.cp_ver); + return le64_to_cpu(CACHED_NODE(entry)->footer.cp_ver); } -static inline block_t next_blkaddr_of_node(const struct folio *node_folio) +static inline block_t next_blkaddr_of_node(const struct f2fs_cached_block *entry) { - struct f2fs_node *rn = F2FS_NODE(node_folio); - return le32_to_cpu(rn->footer.next_blkaddr); + return le32_to_cpu(CACHED_NODE(entry)->footer.next_blkaddr); } -static inline void fill_node_footer(const struct folio *folio, nid_t nid, +static inline void fill_node_footer(struct f2fs_cached_block *entry, nid_t nid, nid_t ino, unsigned int ofs, bool reset) { - struct f2fs_node *rn = F2FS_NODE(folio); + struct f2fs_node *rn = CACHED_NODE(entry); unsigned int old_flag = 0; if (reset) @@ -290,18 +285,16 @@ static inline void fill_node_footer(const struct folio *folio, nid_t nid, (old_flag & OFFSET_BIT_MASK)); } -static inline void copy_node_footer(const struct folio *dst, - const struct folio *src) +static inline void copy_node_footer(struct f2fs_cached_block *dst, + const struct f2fs_cached_block *src) { - struct f2fs_node *src_rn = F2FS_NODE(src); - struct f2fs_node *dst_rn = F2FS_NODE(dst); - memcpy(&dst_rn->footer, &src_rn->footer, sizeof(struct node_footer)); + memcpy(&CACHED_NODE(dst)->footer, &CACHED_NODE(src)->footer, sizeof(struct node_footer)); } -static inline void fill_node_footer_blkaddr(struct folio *folio, block_t blkaddr) +static inline void fill_node_footer_blkaddr(struct f2fs_cached_block *entry, block_t blkaddr) { - struct f2fs_checkpoint *ckpt = F2FS_CKPT(F2FS_F_SB(folio)); - struct f2fs_node *rn = F2FS_NODE(folio); + struct f2fs_node *rn = CACHED_NODE(entry); + struct f2fs_checkpoint *ckpt = F2FS_CKPT(entry->cache->sbi); __u64 cp_ver = cur_cp_version(ckpt); if (__is_set_ckpt_flags(ckpt, CP_CRC_RECOVERY_FLAG)) @@ -311,19 +304,20 @@ static inline void fill_node_footer_blkaddr(struct folio *folio, block_t blkaddr rn->footer.next_blkaddr = cpu_to_le32(blkaddr); } -static inline bool is_recoverable_dnode(struct f2fs_sb_info *sbi, const struct folio *folio) +static inline bool is_recoverable_dnode(struct f2fs_sb_info *sbi, + const struct f2fs_cached_block *entry) { struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); __u64 cp_ver = cur_cp_version(ckpt); /* Don't care crc part, if fsck.f2fs sets it. */ if (__is_set_ckpt_flags(ckpt, CP_NOCRC_RECOVERY_FLAG)) - return (cp_ver << 32) == (cpver_of_node(folio) << 32); + return (cp_ver << 32) == (cpver_of_node(entry) << 32); if (__is_set_ckpt_flags(ckpt, CP_CRC_RECOVERY_FLAG)) cp_ver |= (cur_cp_crc(ckpt) << 32); - return cp_ver == cpver_of_node(folio); + return cp_ver == cpver_of_node(entry); } /* @@ -347,9 +341,9 @@ static inline bool is_recoverable_dnode(struct f2fs_sb_info *sbi, const struct f * `- indirect node ((6 + 2N) + (N - 1)(N + 1)) * `- direct node */ -static inline bool IS_DNODE(const struct folio *node_folio) +static inline bool IS_DNODE(const struct f2fs_cached_block *entry) { - unsigned int ofs = ofs_of_node(node_folio); + unsigned int ofs = ofs_of_node(entry); if (f2fs_has_xattr_block(ofs)) return true; @@ -365,23 +359,23 @@ static inline bool IS_DNODE(const struct folio *node_folio) return true; } -static inline int set_nid(struct folio *folio, int off, nid_t nid, bool i) +static inline bool set_nid(struct f2fs_cached_block *entry, int off, nid_t nid, bool i) { - struct f2fs_node *rn = F2FS_NODE(folio); + struct f2fs_node *rn = CACHED_NODE(entry); + __le32 *addr = i ? &rn->i.i_nid[off - NODE_DIR1_BLOCK] : &rn->in.nid[off]; - f2fs_folio_wait_writeback(folio, NODE, true, true); + f2fs_cache_wait_writeback(entry); + if (*addr == cpu_to_le32(nid)) + return false; - if (i) - rn->i.i_nid[off - NODE_DIR1_BLOCK] = cpu_to_le32(nid); - else - rn->in.nid[off] = cpu_to_le32(nid); - return folio_mark_dirty(folio); + *addr = cpu_to_le32(nid); + f2fs_mark_cache_dirty(entry); + return true; } -static inline nid_t get_nid(const struct folio *folio, int off, bool i) +static inline nid_t get_nid(const struct f2fs_cached_block *entry, int off, bool i) { - struct f2fs_node *rn = F2FS_NODE(folio); - + struct f2fs_node *rn = CACHED_NODE(entry); if (i) return le32_to_cpu(rn->i.i_nid[off - NODE_DIR1_BLOCK]); return le32_to_cpu(rn->in.nid[off]); @@ -394,19 +388,18 @@ static inline nid_t get_nid(const struct folio *folio, int off, bool i) * - Mark cold data pages in page cache */ -static inline int is_node(const struct folio *folio, int type) +static inline int is_node(const struct f2fs_cached_block *entry, int type) { - struct f2fs_node *rn = F2FS_NODE(folio); - return le32_to_cpu(rn->footer.flag) & BIT(type); + return le32_to_cpu(CACHED_NODE(entry)->footer.flag) & BIT(type); } -#define is_cold_node(folio) is_node(folio, COLD_BIT_SHIFT) -#define is_fsync_dnode(folio) is_node(folio, FSYNC_BIT_SHIFT) -#define is_dent_dnode(folio) is_node(folio, DENT_BIT_SHIFT) +#define is_cold_node(entry) is_node(entry, COLD_BIT_SHIFT) +#define is_fsync_dnode(entry) is_node(entry, FSYNC_BIT_SHIFT) +#define is_dent_dnode(entry) is_node(entry, DENT_BIT_SHIFT) -static inline void __set_mark(const struct folio *folio, bool mark, int type) +static inline void __set_mark(struct f2fs_cached_block *entry, bool mark, int type) { - struct f2fs_node *rn = F2FS_NODE(folio); + struct f2fs_node *rn = CACHED_NODE(entry); unsigned int flag = le32_to_cpu(rn->footer.flag); if (mark) @@ -416,18 +409,18 @@ static inline void __set_mark(const struct folio *folio, bool mark, int type) rn->footer.flag = cpu_to_le32(flag); } -static inline void set_cold_node(const struct folio *folio, bool is_dir) +static inline void set_cold_node(struct f2fs_cached_block *entry, bool is_dir) { - __set_mark(folio, !is_dir, COLD_BIT_SHIFT); + __set_mark(entry, !is_dir, COLD_BIT_SHIFT); } -static inline void set_mark(struct folio *folio, bool mark, int type) +static inline void set_mark(struct f2fs_cached_block *entry, bool mark, int type) { - __set_mark(folio, mark, type); - + __set_mark(entry, mark, type); #ifdef CONFIG_F2FS_CHECK_FS - f2fs_inode_chksum_set(F2FS_F_SB(folio), folio); + f2fs_inode_chksum_set(entry->cache->sbi, entry); #endif } -#define set_dentry_mark(folio, mark) set_mark(folio, mark, DENT_BIT_SHIFT) -#define set_fsync_mark(folio, mark) set_mark(folio, mark, FSYNC_BIT_SHIFT) + +#define set_dentry_mark(entry, mark) set_mark(entry, mark, DENT_BIT_SHIFT) +#define set_fsync_mark(entry, mark) set_mark(entry, mark, FSYNC_BIT_SHIFT) diff --git a/fs/f2fs/recovery.c b/fs/f2fs/recovery.c index 5cb56b4c1879..1e414b5a205f 100644 --- a/fs/f2fs/recovery.c +++ b/fs/f2fs/recovery.c @@ -190,7 +190,7 @@ static int recover_dentry(struct inode *inode, struct f2fs_cached_block *entry, struct f2fs_dir_entry *de; struct f2fs_filename fname; struct qstr usr_fname; - struct folio *folio; + void *dentry_blk = NULL; struct inode *dir, *einode; struct fsync_inode_entry *fsync_entry; int err = 0; @@ -213,7 +213,8 @@ static int recover_dentry(struct inode *inode, struct f2fs_cached_block *entry, if (err) goto out; retry: - de = __f2fs_find_entry(dir, &fname, &folio); + dentry_blk = NULL; + de = __f2fs_find_entry(dir, &fname, &dentry_blk); if (de && inode->i_ino == le32_to_cpu(de->ino)) goto out_put; @@ -238,11 +239,11 @@ static int recover_dentry(struct inode *inode, struct f2fs_cached_block *entry, iput(einode); goto out_put; } - f2fs_delete_entry(de, folio, dir, einode); + f2fs_delete_entry(de, dentry_blk, dir, einode); iput(einode); goto retry; - } else if (IS_ERR(folio)) { - err = PTR_ERR(folio); + } else if (IS_ERR(dentry_blk)) { + err = PTR_ERR(dentry_blk); } else { err = f2fs_add_dentry(dir, &fname, inode, inode->i_ino, inode->i_mode); @@ -252,11 +253,11 @@ static int recover_dentry(struct inode *inode, struct f2fs_cached_block *entry, goto out; out_put: - f2fs_folio_put(folio, false); + f2fs_put_dentry_block(dentry_blk, false); out: name = recover_printable_name(inode, raw_inode, &name_len); f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, dir = %llu, err = %d", - __func__, ino_of_node(cache_folio(entry)), name_len, name, + __func__, ino_of_node(entry), name_len, name, IS_ERR(dir) ? 0 : dir->i_ino, err); return err; } @@ -357,7 +358,7 @@ static int recover_inode(struct inode *inode, struct f2fs_cached_block *entry) name = recover_printable_name(inode, raw, &name_len); f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, inline = %x", - __func__, ino_of_node(cache_folio(entry)), name_len, name, + __func__, ino_of_node(entry), name_len, name, raw->i_inline); return 0; } @@ -397,16 +398,16 @@ static int sanity_check_node_chain(struct f2fs_sb_info *sbi, block_t blkaddr, if (IS_ERR(entry)) return PTR_ERR(entry); - if (!is_recoverable_dnode(sbi, cache_folio(entry))) { + if (!is_recoverable_dnode(sbi, entry)) { f2fs_put_cache(entry, true); *is_detecting = false; return 0; } ra_blocks = adjust_por_ra_blocks(sbi, ra_blocks, *blkaddr_fast, - next_blkaddr_of_node(cache_folio(entry))); + next_blkaddr_of_node(entry)); - *blkaddr_fast = next_blkaddr_of_node(cache_folio(entry)); + *blkaddr_fast = next_blkaddr_of_node(entry); f2fs_put_cache(entry, true); f2fs_ra_meta_caches_cond(sbi, *blkaddr_fast, ra_blocks); @@ -448,21 +449,21 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head, } rn = CACHED_NODE(entry); - if (!is_recoverable_dnode(sbi, cache_folio(entry))) { + if (!is_recoverable_dnode(sbi, entry)) { f2fs_put_cache(entry, true); break; } - if (!is_fsync_dnode(cache_folio(entry))) + if (!is_fsync_dnode(entry)) goto next; - fsync_entry = get_fsync_inode(head, ino_of_node(cache_folio(entry))); + fsync_entry = get_fsync_inode(head, ino_of_node(entry)); if (!fsync_entry) { bool quota_inode = false; if (!check_only && - IS_INODE(cache_folio(entry)) && - is_dent_dnode(cache_folio(entry))) { + IS_INODE(entry) && + is_dent_dnode(entry)) { err = f2fs_recover_inode_page(sbi, entry); if (err) { f2fs_put_cache(entry, true); @@ -471,7 +472,7 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head, quota_inode = true; } - fsync_entry = add_fsync_inode(sbi, head, ino_of_node(cache_folio(entry)), + fsync_entry = add_fsync_inode(sbi, head, ino_of_node(entry), quota_inode); if (IS_ERR(fsync_entry)) { err = PTR_ERR(fsync_entry); @@ -490,11 +491,11 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head, } fsync_entry->blkaddr = blkaddr; - if (IS_INODE(cache_folio(entry)) && is_dent_dnode(cache_folio(entry))) + if (IS_INODE(entry) && is_dent_dnode(entry)) fsync_entry->last_dentry = blkaddr; next: /* check next segment */ - blkaddr = next_blkaddr_of_node(cache_folio(entry)); + blkaddr = next_blkaddr_of_node(entry); f2fs_put_cache(entry, true); err = sanity_check_node_chain(sbi, blkaddr, &blkaddr_fast, @@ -522,7 +523,7 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, struct f2fs_summary_block *sum_node; struct f2fs_summary sum; struct f2fs_cached_block *entry = NULL; - struct folio *node_folio; + struct f2fs_cached_block *node_entry; struct dnode_of_data tdn = *dn; nid_t ino, nid; struct inode *inode; @@ -555,7 +556,7 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, nid = le32_to_cpu(sum.nid); ofs_in_node = le16_to_cpu(sum.ofs_in_node); - max_addrs = ADDRS_PER_PAGE(dn->node_folio, dn->inode); + max_addrs = ADDRS_PER_PAGE(dn->node_entry, dn->inode); if (ofs_in_node >= max_addrs) { f2fs_err(sbi, "Inconsistent ofs_in_node:%u in summary, ino:%llu, nid:%u, max:%u", ofs_in_node, dn->inode->i_ino, nid, max_addrs); @@ -565,9 +566,9 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, if (dn->inode->i_ino == nid) { tdn.nid = nid; - if (!dn->inode_folio_locked) - folio_lock(dn->inode_folio); - tdn.node_folio = dn->inode_folio; + if (!dn->inode_entry_locked) + f2fs_lock_cache(dn->inode_entry); + tdn.node_entry = dn->inode_entry; tdn.ofs_in_node = ofs_in_node; goto truncate_out; } else if (dn->nid == nid) { @@ -576,13 +577,13 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, } /* Get the node page */ - node_folio = f2fs_get_node_folio(sbi, nid, NODE_TYPE_REGULAR); - if (IS_ERR(node_folio)) - return PTR_ERR(node_folio); + node_entry = f2fs_get_node_cache(sbi, nid, NODE_TYPE_REGULAR); + if (IS_ERR(node_entry)) + return PTR_ERR(node_entry); - offset = ofs_of_node(node_folio); - ino = ino_of_node(node_folio); - f2fs_folio_put(node_folio, true); + offset = ofs_of_node(node_entry); + ino = ino_of_node(node_entry); + f2fs_put_cache(node_entry, true); if (ino != dn->inode->i_ino) { int ret; @@ -608,8 +609,8 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, * if inode page is locked, unlock temporarily, but its reference * count keeps alive. */ - if (ino == dn->inode->i_ino && dn->inode_folio_locked) - folio_unlock(dn->inode_folio); + if (ino == dn->inode->i_ino && dn->inode_entry_locked) + f2fs_unlock_cache(dn->inode_entry); set_new_dnode(&tdn, inode, NULL, NULL, 0); if (f2fs_get_dnode_of_data(&tdn, bidx, LOOKUP_NODE)) @@ -622,15 +623,15 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi, out: if (ino != dn->inode->i_ino) iput(inode); - else if (dn->inode_folio_locked) - folio_lock(dn->inode_folio); + else if (dn->inode_entry_locked) + f2fs_lock_cache(dn->inode_entry); return 0; truncate_out: if (f2fs_data_blkaddr(&tdn) == blkaddr) f2fs_truncate_data_blocks_range(&tdn, 1); - if (dn->inode->i_ino == nid && !dn->inode_folio_locked) - folio_unlock(dn->inode_folio); + if (dn->inode->i_ino == nid && !dn->inode_entry_locked) + f2fs_unlock_cache(dn->inode_entry); return 0; } @@ -656,11 +657,11 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, int err = 0, recovered = 0; /* step 1: recover xattr */ - if (IS_INODE(cache_folio(entry))) { + if (IS_INODE(entry)) { err = f2fs_recover_inline_xattr(inode, entry); if (err) goto out; - } else if (f2fs_has_xattr_block(ofs_of_node(cache_folio(entry)))) { + } else if (f2fs_has_xattr_block(ofs_of_node(entry))) { err = f2fs_recover_xattr_data(inode, entry); if (!err) recovered++; @@ -676,8 +677,8 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, } /* step 3: recover data indices */ - start = f2fs_start_bidx_of_node(ofs_of_node(cache_folio(entry)), inode); - end = start + addrs_per_page(inode, IS_INODE(cache_folio(entry))); + start = f2fs_start_bidx_of_node(ofs_of_node(entry), inode); + end = start + addrs_per_page(inode, IS_INODE(entry)); set_new_dnode(&dn, inode, NULL, NULL, 0); retry_dn: @@ -690,18 +691,18 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, goto out; } - f2fs_folio_wait_writeback(dn.node_folio, NODE, true, true); + f2fs_cache_wait_writeback(dn.node_entry); err = f2fs_get_node_info(sbi, dn.nid, &ni, false); if (err) goto err; - f2fs_bug_on(sbi, ni.ino != ino_of_node(cache_folio(entry))); + f2fs_bug_on(sbi, ni.ino != ino_of_node(entry)); - if (ofs_of_node(dn.node_folio) != ofs_of_node(cache_folio(entry))) { + if (ofs_of_node(dn.node_entry) != ofs_of_node(entry)) { f2fs_warn(sbi, "Inconsistent ofs_of_node, ino:%llu, ofs:%u, %u", - inode->i_ino, ofs_of_node(dn.node_folio), - ofs_of_node(cache_folio(entry))); + inode->i_ino, ofs_of_node(dn.node_entry), + ofs_of_node(entry)); err = -EFSCORRUPTED; f2fs_handle_error(sbi, ERROR_INCONSISTENT_FOOTER); fserror_report_file_metadata(dn.inode, err, GFP_NOFS); @@ -712,7 +713,7 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, block_t src, dest; src = f2fs_data_blkaddr(&dn); - dest = data_blkaddr(dn.inode, cache_folio(entry), dn.ofs_in_node); + dest = data_blkaddr(dn.inode, entry, dn.ofs_in_node); if (__is_valid_data_blkaddr(src) && !f2fs_is_valid_blkaddr(sbi, src, META_POR)) { @@ -787,16 +788,16 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, } } - copy_node_footer(dn.node_folio, cache_folio(entry)); - fill_node_footer(dn.node_folio, dn.nid, ni.ino, - ofs_of_node(cache_folio(entry)), false); - folio_mark_dirty(dn.node_folio); + copy_node_footer(dn.node_entry, entry); + fill_node_footer(dn.node_entry, dn.nid, ni.ino, + ofs_of_node(entry), false); + f2fs_mark_cache_dirty(dn.node_entry); err: f2fs_put_dnode(&dn); out: f2fs_notice(sbi, "recover_data: ino = %llx, nid = %x (i_size: %s), " "range (%u, %u), recovered = %d, err = %d", - inode->i_ino, nid_of_node(cache_folio(entry)), + inode->i_ino, nid_of_node(entry), file_keep_isize(inode) ? "keep" : "recover", start, end, recovered, err); return err; @@ -835,13 +836,13 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list, break; } - if (!is_recoverable_dnode(sbi, cache_folio(entry))) { + if (!is_recoverable_dnode(sbi, entry)) { f2fs_put_cache(entry, true); break; } recoverable_dnode++; - fsync_entry = get_fsync_inode(inode_list, ino_of_node(cache_folio(entry))); + fsync_entry = get_fsync_inode(inode_list, ino_of_node(entry)); if (!fsync_entry) goto next; fsynced_dnode++; @@ -850,7 +851,7 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list, * In this case, we can lose the latest inode(x). * So, call recover_inode for the inode update. */ - if (IS_INODE(cache_folio(entry))) { + if (IS_INODE(entry)) { err = recover_inode(fsync_entry->inode, entry); if (err) { f2fs_put_cache(entry, true); @@ -877,10 +878,10 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list, list_move_tail(&fsync_entry->list, tmp_inode_list); next: ra_blocks = adjust_por_ra_blocks(sbi, ra_blocks, blkaddr, - next_blkaddr_of_node(cache_folio(entry))); + next_blkaddr_of_node(entry)); /* check next segment */ - blkaddr = next_blkaddr_of_node(cache_folio(entry)); + blkaddr = next_blkaddr_of_node(entry); f2fs_put_cache(entry, true); f2fs_ra_meta_caches_cond(sbi, blkaddr, ra_blocks); @@ -943,7 +944,7 @@ int f2fs_recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only) f2fs_truncate_meta_caches(sbi, MAIN_BLKADDR(sbi), MAX_BLKADDR(sbi) - MAIN_BLKADDR(sbi)); if (err) { - truncate_inode_pages_final(NODE_MAPPING(sbi)); + f2fs_truncate_node_caches(sbi, 0, ULONG_MAX); f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); } diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c index cbb3a8c9f4ab..3a585fc7c197 100644 --- a/fs/f2fs/segment.c +++ b/fs/f2fs/segment.c @@ -335,7 +335,7 @@ static int __f2fs_commit_atomic_write(struct inode *inode) goto next; } - blen = min((pgoff_t)ADDRS_PER_PAGE(dn.node_folio, cow_inode), + blen = min((pgoff_t)addrs_per_page(cow_inode, IS_INODE(dn.node_entry)), len); index = off; for (i = 0; i < blen; i++, dn.ofs_in_node++, index++) { @@ -3768,7 +3768,8 @@ static int __get_segment_type_4(struct f2fs_io_info *fio) else return CURSEG_COLD_DATA; } else { - if (IS_DNODE(fio->folio) && is_cold_node(fio->folio)) + f2fs_bug_on(fio->sbi, !fio->is_cache); + if (IS_DNODE(fio->cache_entry) && is_cold_node(fio->cache_entry)) return CURSEG_WARM_NODE; else return CURSEG_COLD_NODE; @@ -3826,9 +3827,9 @@ static int __get_segment_type_6(struct f2fs_io_info *fio) return f2fs_rw_hint_to_seg_type(F2FS_I_SB(inode), inode->i_write_hint); } else { - if (IS_DNODE(fio->folio)) - return is_cold_node(fio->folio) ? CURSEG_WARM_NODE : - CURSEG_HOT_NODE; + f2fs_bug_on(fio->sbi, !fio->is_cache); + if (IS_DNODE(fio->cache_entry)) + return is_cold_node(fio->cache_entry) ? CURSEG_WARM_NODE : CURSEG_HOT_NODE; return CURSEG_COLD_NODE; } } @@ -3895,7 +3896,7 @@ static void f2fs_randomize_chunk(struct f2fs_sb_info *sbi, get_random_u32_inclusive(1, sbi->max_fragment_hole); } -int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, struct folio *folio, +int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, block_t old_blkaddr, block_t *new_blkaddr, struct f2fs_summary *sum, int type, struct f2fs_io_info *fio) @@ -4003,10 +4004,10 @@ int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, struct folio *folio, up_write(&sit_i->sentry_lock); - if (folio && IS_NODESEG(curseg->seg_type)) { - fill_node_footer_blkaddr(folio, NEXT_FREE_BLKADDR(sbi, curseg)); - - f2fs_inode_chksum_set(sbi, folio); + if (fio && fio->is_cache && IS_NODESEG(curseg->seg_type)) { + fill_node_footer_blkaddr(fio->cache_entry, + NEXT_FREE_BLKADDR(sbi, curseg)); + f2fs_inode_chksum_set(sbi, fio->cache_entry); } if (fio) { @@ -4094,18 +4095,26 @@ static void do_write_page(struct f2fs_summary *sum, struct f2fs_io_info *fio) if (keep_order) f2fs_down_read(&fio->sbi->io_order_lock); - err = f2fs_allocate_data_block(fio->sbi, folio, fio->old_blkaddr, + err = f2fs_allocate_data_block(fio->sbi, fio->old_blkaddr, &fio->new_blkaddr, sum, type, fio); if (unlikely(err)) { - f2fs_err_ratelimited(fio->sbi, - "%s Failed to allocate data block, ino:%u, index:%lu, type:%d, old_blkaddr:0x%x, new_blkaddr:0x%x, err:%d", - __func__, fio->ino, folio->index, type, - fio->old_blkaddr, fio->new_blkaddr, err); - if (fscrypt_inode_uses_fs_layer_crypto(folio->mapping->host)) - fscrypt_finalize_bounce_page(&fio->encrypted_page); - folio_end_writeback(folio); - if (f2fs_in_warm_node_list(folio)) - f2fs_del_fsync_node_entry(fio->sbi, folio); + if (fio->is_cache) { + f2fs_err_ratelimited(fio->sbi, + "%s Failed to allocate data block, ino:%u, index:%lu, type:%d, old_blkaddr:0x%x, new_blkaddr:0x%x, err:%d", + __func__, fio->ino, fio->cache_entry->index, type, + fio->old_blkaddr, fio->new_blkaddr, err); + f2fs_end_cache_writeback(fio->cache_entry); + if (f2fs_in_warm_node_list(fio->cache_entry)) + f2fs_del_fsync_node_entry(fio->sbi, fio->cache_entry); + } else { + f2fs_err_ratelimited(fio->sbi, + "%s Failed to allocate data block, ino:%u, index:%lu, type:%d, old_blkaddr:0x%x, new_blkaddr:0x%x, err:%d", + __func__, fio->ino, folio->index, type, + fio->old_blkaddr, fio->new_blkaddr, err); + if (fscrypt_inode_uses_fs_layer_crypto(folio->mapping->host)) + fscrypt_finalize_bounce_page(&fio->encrypted_page); + folio_end_writeback(folio); + } f2fs_bug_on(fio->sbi, !is_set_ckpt_flags(fio->sbi, CP_ERROR_FLAG)); goto out; @@ -4118,7 +4127,10 @@ static void do_write_page(struct f2fs_summary *sum, struct f2fs_io_info *fio) f2fs_invalidate_internal_cache(fio->sbi, fio->old_blkaddr, 1); /* writeout dirty page into bdev */ - f2fs_submit_page_write(fio); + if (fio->is_cache) + f2fs_submit_cache_write(fio); + else + f2fs_submit_page_write(fio); f2fs_update_device_state(fio->sbi, fio->ino, fio->new_blkaddr, 1); out: @@ -4351,14 +4363,13 @@ void f2fs_replace_block(struct f2fs_sb_info *sbi, struct dnode_of_data *dn, f2fs_update_data_blkaddr(dn, new_addr); } -void f2fs_folio_wait_writeback(struct folio *folio, enum page_type type, - bool ordered, bool locked) +void f2fs_folio_wait_writeback(struct folio *folio, bool ordered, bool locked) { if (folio_test_writeback(folio)) { struct f2fs_sb_info *sbi = F2FS_F_SB(folio); /* submit cached LFS IO */ - f2fs_submit_merged_write_folio(sbi, folio, type); + f2fs_submit_merged_write_folio(sbi, folio); /* submit cached IPU IO */ f2fs_submit_merged_ipu_write(sbi, NULL, folio); if (ordered) { diff --git a/fs/f2fs/segment.h b/fs/f2fs/segment.h index fff35c63a00f..341939478fd4 100644 --- a/fs/f2fs/segment.h +++ b/fs/f2fs/segment.h @@ -1014,26 +1014,6 @@ static inline long adjust_flush_cache_number(struct f2fs_sb_info *sbi, int type) return nr_to_write; } -/* - * When writing pages, it'd better align nr_to_write for segment size. - */ -static inline long nr_pages_to_write(struct f2fs_sb_info *sbi, int type, - struct writeback_control *wbc) -{ - long nr_to_write, desired; - - if (wbc->sync_mode != WB_SYNC_NONE) - return 0; - - nr_to_write = wbc->nr_to_write; - desired = BIO_MAX_VECS; - if (type == NODE) - desired <<= 1; - - wbc->nr_to_write = desired; - return desired - nr_to_write; -} - static inline void wake_up_discard_thread(struct f2fs_sb_info *sbi, bool force) { struct discard_cmd_control *dcc = SM_I(sbi)->dcc_info; diff --git a/fs/f2fs/shrinker.c b/fs/f2fs/shrinker.c index 1755c85849e4..20b3fe1f8c07 100644 --- a/fs/f2fs/shrinker.c +++ b/fs/f2fs/shrinker.c @@ -39,7 +39,8 @@ static unsigned long __count_extent_cache(struct f2fs_sb_info *sbi, static unsigned long __count_cache(struct f2fs_sb_info *sbi) { - return sbi->meta_blocks.num_entries; + return sbi->meta_blocks.num_entries + + sbi->node_blocks.num_entries; } unsigned long f2fs_shrink_count(struct shrinker *shrink, diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index 83496c46c89f..bc1cff6cad0d 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -1831,20 +1831,8 @@ static struct inode *f2fs_alloc_inode(struct super_block *sb) static int f2fs_drop_inode(struct inode *inode) { - struct f2fs_sb_info *sbi = F2FS_I_SB(inode); int ret; - /* - * during filesystem shutdown, if checkpoint is disabled, - * drop useless meta/node dirty pages. - */ - if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) { - if (inode->i_ino == F2FS_NODE_INO(sbi)) { - trace_f2fs_drop_inode(inode, 1); - return 1; - } - } - /* * This is to avoid a deadlock condition like below. * writeback_single_inode(inode) @@ -1865,7 +1853,7 @@ static int f2fs_drop_inode(struct inode *inode) f2fs_i_size_write(inode, 0); f2fs_submit_merged_write_cond(F2FS_I_SB(inode), - inode, NULL, 0, DATA); + inode, NULL); truncate_inode_pages_final(inode->i_mapping); if (F2FS_HAS_BLOCKS(inode)) @@ -1939,11 +1927,6 @@ void f2fs_inode_synced(struct inode *inode) */ static void f2fs_dirty_inode(struct inode *inode, int flags) { - struct f2fs_sb_info *sbi = F2FS_I_SB(inode); - - if (inode->i_ino == F2FS_NODE_INO(sbi)) - return; - if (is_inode_flag_set(inode, FI_AUTO_RECOVER)) clear_inode_flag(inode, FI_AUTO_RECOVER); @@ -2039,7 +2022,7 @@ static void f2fs_put_super(struct super_block *sb) if (err || f2fs_cp_error(sbi) || unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) { - truncate_inode_pages_final(NODE_MAPPING(sbi)); + f2fs_truncate_node_caches(sbi, 0, ULONG_MAX); f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); } @@ -2047,11 +2030,8 @@ static void f2fs_put_super(struct super_block *sb) f2fs_destroy_compress_inode(sbi); - iput(sbi->node_inode); - sbi->node_inode = NULL; - - f2fs_destroy_cache(META_CACHE(sbi)); f2fs_destroy_cache(NODE_CACHE(sbi)); + f2fs_destroy_cache(META_CACHE(sbi)); /* Should check the page counts after dropping all node/meta pages */ for (i = 0; i < NR_COUNT_TYPE; i++) { @@ -4379,7 +4359,6 @@ static void init_sb_info(struct f2fs_sb_info *sbi) sbi->allocate_section_hint = le32_to_cpu(raw_super->section_count); sbi->allocate_section_policy = ALLOCATE_FORWARD_NOHINT; F2FS_ROOT_INO(sbi) = le32_to_cpu(raw_super->root_ino); - F2FS_NODE_INO(sbi) = le32_to_cpu(raw_super->node_ino); sbi->cur_victim_sec = NULL_SECNO; sbi->gc_mode = GC_NORMAL; sbi->next_victim_seg[BG_GC] = NULL_SEGNO; @@ -5030,7 +5009,7 @@ static void f2fs_restore_device_alias(struct f2fs_sb_info *sbi) { struct inode *root = d_inode(sbi->sb->s_root); struct f2fs_dir_entry *de; - struct folio *folio; + void *dentry_block = NULL; int i; if (!f2fs_sb_has_device_alias(sbi)) @@ -5045,7 +5024,7 @@ static void f2fs_restore_device_alias(struct f2fs_sb_info *sbi) qstr.name = name; qstr.len = strlen(name); - de = f2fs_find_entry(root, &qstr, &folio); + de = f2fs_find_entry(root, &qstr, &dentry_block); if (!de) continue; @@ -5055,7 +5034,7 @@ static void f2fs_restore_device_alias(struct f2fs_sb_info *sbi) FDEV(i).has_alias = true; iput(inode); } - f2fs_folio_put(folio, 0); + f2fs_put_dentry_block(dentry_block, false); } } @@ -5322,33 +5301,25 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) if (err) goto free_nm; - /* get an inode for node space */ - sbi->node_inode = f2fs_iget(sb, F2FS_NODE_INO(sbi)); - if (IS_ERR(sbi->node_inode)) { - f2fs_err(sbi, "Failed to read node inode"); - err = PTR_ERR(sbi->node_inode); - goto free_stats; - } - /* read root inode and dentry */ root = f2fs_iget(sb, F2FS_ROOT_INO(sbi)); if (IS_ERR(root)) { f2fs_err(sbi, "Failed to read root inode"); err = PTR_ERR(root); - goto free_node_inode; + goto free_ino_entry; } if (!S_ISDIR(root->i_mode) || !root->i_blocks || !root->i_size || !root->i_nlink) { iput(root); err = -EINVAL; - goto free_node_inode; + goto free_ino_entry; } generic_set_sb_d_ops(sb); sb->s_root = d_make_root(root); /* allocate root dentry */ if (!sb->s_root) { err = -ENOMEM; - goto free_node_inode; + goto free_ino_entry; } err = f2fs_init_compress_inode(sbi); @@ -5519,7 +5490,7 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) * Some dirty meta pages can be produced by f2fs_recover_orphan_inodes() * failed by EIO. Then, iput(node_inode) can trigger balance_fs_bg() * followed by f2fs_write_checkpoint() through f2fs_write_node_pages(), which - * falls into an infinite loop in f2fs_sync_meta_pages(). + * falls into an infinite loop in f2fs_sync_meta_caches(). */ f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX); /* evict some inodes being cached by GC */ @@ -5530,12 +5501,9 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) free_root_inode: dput(sb->s_root); sb->s_root = NULL; -free_node_inode: +free_ino_entry: f2fs_release_ino_entry(sbi, true); - truncate_inode_pages_final(NODE_MAPPING(sbi)); - iput(sbi->node_inode); - sbi->node_inode = NULL; -free_stats: + f2fs_truncate_node_caches(sbi, 0, ULONG_MAX); f2fs_destroy_stats(sbi); free_nm: /* stop discard thread before destroying node manager */ diff --git a/fs/f2fs/xattr.c b/fs/f2fs/xattr.c index 6728d1488cad..3169e361ea68 100644 --- a/fs/f2fs/xattr.c +++ b/fs/f2fs/xattr.c @@ -138,7 +138,7 @@ static int f2fs_xattr_advise_set(const struct xattr_handler *handler, #ifdef CONFIG_F2FS_FS_SECURITY static int f2fs_initxattrs(struct inode *inode, const struct xattr *xattr_array, - void *folio) + void *fs_data) { const struct xattr *xattr; int err = 0; @@ -146,7 +146,7 @@ static int f2fs_initxattrs(struct inode *inode, const struct xattr *xattr_array, for (xattr = xattr_array; xattr->name != NULL; xattr++) { err = f2fs_setxattr(inode, F2FS_XATTR_INDEX_SECURITY, xattr->name, xattr->value, - xattr->value_len, folio, 0); + xattr->value_len, fs_data, 0); if (err < 0) break; } @@ -154,10 +154,10 @@ static int f2fs_initxattrs(struct inode *inode, const struct xattr *xattr_array, } int f2fs_init_security(struct inode *inode, struct inode *dir, - const struct qstr *qstr, struct folio *ifolio) + const struct qstr *qstr, struct f2fs_cached_block *ientry) { return security_inode_init_security(inode, dir, qstr, - f2fs_initxattrs, ifolio); + f2fs_initxattrs, ientry); } #endif @@ -273,25 +273,25 @@ static struct f2fs_xattr_entry *__find_inline_xattr(struct inode *inode, return entry; } -static int read_inline_xattr(struct inode *inode, struct folio *ifolio, +static int read_inline_xattr(struct inode *inode, struct f2fs_cached_block *ientry, void *txattr_addr) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); unsigned int inline_size = inline_xattr_size(inode); - struct folio *folio = NULL; + struct f2fs_cached_block *in_entry = NULL; void *inline_addr; - if (ifolio) { - inline_addr = inline_xattr_addr(inode, ifolio); + if (ientry) { + inline_addr = inline_xattr_addr(inode, ientry); } else { - folio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(folio)) - return PTR_ERR(folio); + in_entry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(in_entry)) + return PTR_ERR(in_entry); - inline_addr = inline_xattr_addr(inode, folio); + inline_addr = inline_xattr_addr(inode, in_entry); } memcpy(txattr_addr, inline_addr, inline_size); - f2fs_folio_put(folio, true); + f2fs_put_cache(in_entry, true); return 0; } @@ -301,22 +301,22 @@ static int read_xattr_block(struct inode *inode, void *txattr_addr) struct f2fs_sb_info *sbi = F2FS_I_SB(inode); nid_t xnid = F2FS_I(inode)->i_xattr_nid; unsigned int inline_size = inline_xattr_size(inode); - struct folio *xfolio; + struct f2fs_cached_block *xentry; void *xattr_addr; /* The inode already has an extended attribute block. */ - xfolio = f2fs_get_xnode_folio(sbi, xnid); - if (IS_ERR(xfolio)) - return PTR_ERR(xfolio); + xentry = f2fs_get_xnode_cache(sbi, xnid); + if (IS_ERR(xentry)) + return PTR_ERR(xentry); - xattr_addr = folio_address(xfolio); + xattr_addr = cache_address(xentry); memcpy(txattr_addr + inline_size, xattr_addr, VALID_XATTR_BLOCK_SIZE); - f2fs_folio_put(xfolio, true); + f2fs_put_cache(xentry, true); return 0; } -static int lookup_all_xattrs(struct inode *inode, struct folio *ifolio, +static int lookup_all_xattrs(struct inode *inode, struct f2fs_cached_block *ientry, unsigned int index, unsigned int len, const char *name, struct f2fs_xattr_entry **xe, void **base_addr, int *base_size, @@ -340,7 +340,7 @@ static int lookup_all_xattrs(struct inode *inode, struct folio *ifolio, /* read from inline xattr */ if (inline_size) { - err = read_inline_xattr(inode, ifolio, txattr_addr); + err = read_inline_xattr(inode, ientry, txattr_addr); if (err) goto out; @@ -388,7 +388,7 @@ static int lookup_all_xattrs(struct inode *inode, struct folio *ifolio, return err; } -static int read_all_xattrs(struct inode *inode, struct folio *ifolio, +static int read_all_xattrs(struct inode *inode, struct f2fs_cached_block *ientry, void **base_addr) { struct f2fs_xattr_header *header; @@ -405,7 +405,7 @@ static int read_all_xattrs(struct inode *inode, struct folio *ifolio, /* read from inline xattr */ if (inline_size) { - err = read_inline_xattr(inode, ifolio, txattr_addr); + err = read_inline_xattr(inode, ientry, txattr_addr); if (err) goto fail; } @@ -432,14 +432,14 @@ static int read_all_xattrs(struct inode *inode, struct folio *ifolio, } static inline int write_all_xattrs(struct inode *inode, __u32 hsize, - void *txattr_addr, struct folio *ifolio) + void *txattr_addr, struct f2fs_cached_block *ientry) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); size_t inline_size = inline_xattr_size(inode); - struct folio *in_folio = NULL; + struct f2fs_cached_block *in_entry = NULL; void *xattr_addr; void *inline_addr = NULL; - struct folio *xfolio; + struct f2fs_cached_block *xentry; nid_t new_nid = 0; int err = 0; @@ -449,75 +449,74 @@ static inline int write_all_xattrs(struct inode *inode, __u32 hsize, /* write to inline xattr */ if (inline_size) { - if (ifolio) { - inline_addr = inline_xattr_addr(inode, ifolio); + if (ientry) { + inline_addr = inline_xattr_addr(inode, ientry); } else { - in_folio = f2fs_get_inode_folio(sbi, inode->i_ino); - if (IS_ERR(in_folio)) { + in_entry = f2fs_get_inode_cache(sbi, inode->i_ino); + if (IS_ERR(in_entry)) { f2fs_alloc_nid_failed(sbi, new_nid); - return PTR_ERR(in_folio); + return PTR_ERR(in_entry); } - inline_addr = inline_xattr_addr(inode, in_folio); + inline_addr = inline_xattr_addr(inode, in_entry); } - f2fs_folio_wait_writeback(ifolio ? ifolio : in_folio, - NODE, true, true); + f2fs_cache_wait_writeback(ientry ? ientry : in_entry); /* no need to use xattr node block */ if (hsize <= inline_size) { err = f2fs_truncate_xattr_node(inode); f2fs_alloc_nid_failed(sbi, new_nid); if (err) { - f2fs_folio_put(in_folio, true); + f2fs_put_cache(in_entry, true); return err; } memcpy(inline_addr, txattr_addr, inline_size); - folio_mark_dirty(ifolio ? ifolio : in_folio); + f2fs_mark_cache_dirty(ientry ? ientry : in_entry); goto in_page_out; } } /* write to xattr node block */ if (F2FS_I(inode)->i_xattr_nid) { - xfolio = f2fs_get_xnode_folio(sbi, F2FS_I(inode)->i_xattr_nid); - if (IS_ERR(xfolio)) { - err = PTR_ERR(xfolio); + xentry = f2fs_get_xnode_cache(sbi, F2FS_I(inode)->i_xattr_nid); + if (IS_ERR(xentry)) { + err = PTR_ERR(xentry); f2fs_alloc_nid_failed(sbi, new_nid); goto in_page_out; } f2fs_bug_on(sbi, new_nid); - f2fs_folio_wait_writeback(xfolio, NODE, true, true); + f2fs_cache_wait_writeback(xentry); } else { struct dnode_of_data dn; set_new_dnode(&dn, inode, NULL, NULL, new_nid); - xfolio = f2fs_new_node_folio(&dn, XATTR_NODE_OFFSET); - if (IS_ERR(xfolio)) { - err = PTR_ERR(xfolio); + xentry = f2fs_new_node_cache(&dn, XATTR_NODE_OFFSET); + if (IS_ERR(xentry)) { + err = PTR_ERR(xentry); f2fs_alloc_nid_failed(sbi, new_nid); goto in_page_out; } f2fs_alloc_nid_done(sbi, new_nid); } - xattr_addr = folio_address(xfolio); + xattr_addr = cache_address(xentry); if (inline_size) memcpy(inline_addr, txattr_addr, inline_size); memcpy(xattr_addr, txattr_addr + inline_size, VALID_XATTR_BLOCK_SIZE); if (inline_size) - folio_mark_dirty(ifolio ? ifolio : in_folio); - folio_mark_dirty(xfolio); + f2fs_mark_cache_dirty(ientry ? ientry : in_entry); + f2fs_mark_cache_dirty(xentry); - f2fs_folio_put(xfolio, true); + f2fs_put_cache(xentry, true); in_page_out: - f2fs_folio_put(in_folio, true); + f2fs_put_cache(in_entry, true); return err; } int f2fs_getxattr(struct inode *inode, int index, const char *name, - void *buffer, size_t buffer_size, struct folio *ifolio) + void *buffer, size_t buffer_size, struct f2fs_cached_block *ientry) { - struct f2fs_xattr_entry *entry = NULL; + struct f2fs_xattr_entry *xe = NULL; int error; unsigned int size, len; void *base_addr = NULL; @@ -531,16 +530,16 @@ int f2fs_getxattr(struct inode *inode, int index, const char *name, if (len > F2FS_NAME_LEN) return -ERANGE; - if (!ifolio) + if (!ientry) f2fs_down_read(&F2FS_I(inode)->i_xattr_sem); - error = lookup_all_xattrs(inode, ifolio, index, len, name, - &entry, &base_addr, &base_size, &is_inline); - if (!ifolio) + error = lookup_all_xattrs(inode, ientry, index, len, name, + &xe, &base_addr, &base_size, &is_inline); + if (!ientry) f2fs_up_read(&F2FS_I(inode)->i_xattr_sem); if (error) return error; - size = le16_to_cpu(entry->e_value_size); + size = le16_to_cpu(xe->e_value_size); if (buffer && size > buffer_size) { error = -ERANGE; @@ -548,7 +547,7 @@ int f2fs_getxattr(struct inode *inode, int index, const char *name, } if (buffer) { - char *pval = entry->e_name + entry->e_name_len; + char *pval = xe->e_name + xe->e_name_len; if (base_size - (pval - (char *)base_addr) < size) { error = -ERANGE; @@ -632,7 +631,7 @@ static bool f2fs_xattr_value_same(struct f2fs_xattr_entry *entry, static int __f2fs_setxattr(struct inode *inode, int index, const char *name, const void *value, size_t size, - struct folio *ifolio, int flags) + struct f2fs_cached_block *ientry, int flags) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct f2fs_xattr_entry *here, *last; @@ -656,7 +655,7 @@ static int __f2fs_setxattr(struct inode *inode, int index, if (size > MAX_VALUE_LEN(inode)) return -E2BIG; retry: - error = read_all_xattrs(inode, ifolio, &base_addr); + error = read_all_xattrs(inode, ientry, &base_addr); if (error) return error; @@ -773,7 +772,7 @@ static int __f2fs_setxattr(struct inode *inode, int index, *(u32 *)((u8 *)last + newsize) = 0; } - error = write_all_xattrs(inode, new_hsize, base_addr, ifolio); + error = write_all_xattrs(inode, new_hsize, base_addr, ientry); if (error) goto exit; @@ -807,7 +806,7 @@ static int __f2fs_setxattr(struct inode *inode, int index, int f2fs_setxattr(struct inode *inode, int index, const char *name, const void *value, size_t size, - struct folio *ifolio, int flags) + struct f2fs_cached_block *ientry, int flags) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct f2fs_lock_context lc; @@ -823,9 +822,9 @@ int f2fs_setxattr(struct inode *inode, int index, const char *name, return err; /* this case is only from f2fs_init_inode_metadata */ - if (ifolio) + if (ientry) return __f2fs_setxattr(inode, index, name, value, - size, ifolio, flags); + size, ientry, flags); f2fs_balance_fs(sbi, true); f2fs_lock_op(sbi, &lc); diff --git a/fs/f2fs/xattr.h b/fs/f2fs/xattr.h index bce3d93e4755..f50ed518c5cb 100644 --- a/fs/f2fs/xattr.h +++ b/fs/f2fs/xattr.h @@ -130,9 +130,9 @@ extern const struct xattr_handler f2fs_xattr_security_handler; extern const struct xattr_handler * const f2fs_xattr_handlers[]; int f2fs_setxattr(struct inode *, int, const char *, const void *, - size_t, struct folio *, int); + size_t, struct f2fs_cached_block *, int); int f2fs_getxattr(struct inode *, int, const char *, void *, - size_t, struct folio *); + size_t, struct f2fs_cached_block *); ssize_t f2fs_listxattr(struct dentry *, char *, size_t); int __init f2fs_init_xattr_cache(void); void f2fs_destroy_xattr_cache(void); @@ -142,13 +142,13 @@ void f2fs_destroy_xattr_cache(void); #define f2fs_listxattr NULL static inline int f2fs_setxattr(struct inode *inode, int index, const char *name, const void *value, size_t size, - struct folio *folio, int flags) + struct f2fs_cached_block *ientry, int flags) { return -EOPNOTSUPP; } static inline int f2fs_getxattr(struct inode *inode, int index, const char *name, void *buffer, - size_t buffer_size, struct folio *dfolio) + size_t buffer_size, struct f2fs_cached_block *ientry) { return -EOPNOTSUPP; } @@ -158,10 +158,10 @@ static inline void f2fs_destroy_xattr_cache(void) { } #ifdef CONFIG_F2FS_FS_SECURITY int f2fs_init_security(struct inode *, struct inode *, - const struct qstr *, struct folio *); + const struct qstr *, struct f2fs_cached_block *); #else static inline int f2fs_init_security(struct inode *inode, struct inode *dir, - const struct qstr *qstr, struct folio *ifolio) + const struct qstr *qstr, struct f2fs_cached_block *ientry) { return 0; } diff --git a/include/linux/f2fs_fs.h b/include/linux/f2fs_fs.h index 0c027d00a1ea..105cfeedea74 100644 --- a/include/linux/f2fs_fs.h +++ b/include/linux/f2fs_fs.h @@ -34,7 +34,6 @@ #define F2FS_RESERVED_NODE_NUM 3 #define F2FS_ROOT_INO(sbi) ((sbi)->root_ino_num) -#define F2FS_NODE_INO(sbi) ((sbi)->node_ino_num) #define F2FS_COMPRESS_INO(sbi) (NM_I(sbi)->max_nid) #define F2FS_MAX_QUOTAS 3 -- 2.49.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [PATCH v1 08/12] f2fs: cache: initialize compress cache 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel -1 siblings, 0 replies; 34+ messages in thread From: Chao Yu @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-f2fs-devel, linux-kernel, Chao Yu This patch introduces compress_blocks in f2fs_sb_info structure, initializes and destroys the compress cache during filesystem mount and unmount. It adds an ino union field in struct f2fs_cached_block (sharing space with writeback linkage for zero memory overhead) to track per-inode cached blocks, and registers compress cache into the memory shrinker. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 9 ++++++++- fs/f2fs/cache.h | 5 ++++- fs/f2fs/f2fs.h | 6 ++++++ fs/f2fs/super.c | 9 ++++++++- 4 files changed, 26 insertions(+), 3 deletions(-) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index 3176d62ce25b..2cecf6287207 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -194,7 +194,10 @@ static struct f2fs_cached_block *f2fs_create_cache( entry->index = index; atomic_set(&entry->refcount, 0); - entry->next_entry = NULL; + if (!IS_COMPRESS_CACHE(cache)) + entry->next_entry = NULL; + else + entry->ino = 0; INIT_LIST_HEAD(&entry->list); entry->cache = cache; @@ -616,6 +619,10 @@ unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, return freed; freed += f2fs_do_shrink_cache(NODE_CACHE(sbi), nr_to_scan - freed); + if (freed >= nr_to_scan) + return freed; + + freed += f2fs_do_shrink_cache(COMPRESS_CACHE(sbi), nr_to_scan - freed); return freed; } diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index 397019cfb861..23583a52b8f8 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -38,6 +38,7 @@ struct f2fs_sb_info; enum f2fs_cache_type { F2FS_META_CACHE, F2FS_NODE_CACHE, + F2FS_COMPRESS_CACHE, }; /* Main cache control structure (per sb_info) */ @@ -47,12 +48,13 @@ struct f2fs_cached_block_list { spinlock_t tree_lock; /* Lock for radix tree */ struct list_head lru_list; /* Single global LRU list */ spinlock_t list_lock; /* Lock for LRU list */ - enum f2fs_cache_type type; /* Cache type (Node or Meta) */ + enum f2fs_cache_type type; /* Cache type (Node, Meta, Compress) */ unsigned long num_entries; /* Current number of entries */ }; #define IS_META_CACHE(cache) (cache->type == F2FS_META_CACHE) #define IS_NODE_CACHE(cache) (cache->type == F2FS_NODE_CACHE) +#define IS_COMPRESS_CACHE(cache) (cache->type == F2FS_COMPRESS_CACHE) /* Flags for f2fs_cached_block state */ enum f2fs_cached_state { @@ -182,6 +184,7 @@ void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); #define META_CACHE(sbi) (&(sbi)->meta_blocks) #define NODE_CACHE(sbi) (&(sbi)->node_blocks) +#define COMPRESS_CACHE(sbi) (&(sbi)->compress_blocks) #define f2fs_find_meta_cache(sbi, blkaddr) \ f2fs_find_cache(META_CACHE(sbi), blkaddr) diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 4dd165ac05c9..36a6afbf3ba1 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -2104,6 +2104,7 @@ struct f2fs_sb_info { /* f2fs internal cache */ struct f2fs_cached_block_list meta_blocks; struct f2fs_cached_block_list node_blocks; + struct f2fs_cached_block_list compress_blocks; /* internal cache flush thread */ struct f2fs_cache_kthread cache_thread; @@ -2325,6 +2326,11 @@ static inline bool f2fs_is_node_cache(struct f2fs_cached_block *entry) return entry->cache && entry->cache == NODE_CACHE(entry->cache->sbi); } +static inline bool f2fs_is_compress_cache(struct f2fs_cached_block *entry) +{ + return entry->cache && entry->cache == COMPRESS_CACHE(entry->cache->sbi); +} + static inline struct f2fs_bio *F2FS_BIO(struct bio *bio) { return container_of(bio, struct f2fs_bio, bio); diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index bc1cff6cad0d..ac3a94736a91 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -2030,6 +2030,7 @@ static void f2fs_put_super(struct super_block *sb) f2fs_destroy_compress_inode(sbi); + f2fs_destroy_cache(COMPRESS_CACHE(sbi)); f2fs_destroy_cache(NODE_CACHE(sbi)); f2fs_destroy_cache(META_CACHE(sbi)); @@ -5203,10 +5204,14 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) if (err) goto free_meta_cache; + err = f2fs_init_cache(sbi, COMPRESS_CACHE(sbi), F2FS_COMPRESS_CACHE); + if (err) + goto free_node_cache; + err = f2fs_get_valid_checkpoint(sbi); if (err) { f2fs_err(sbi, "Failed to get valid F2FS checkpoint"); - goto free_node_cache; + goto free_compress_cache; } if (__is_set_ckpt_flags(F2FS_CKPT(sbi), CP_QUOTA_NEED_FSCK_FLAG)) @@ -5519,6 +5524,8 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) free_devices: destroy_device_list(sbi); kvfree(sbi->ckpt); +free_compress_cache: + f2fs_destroy_cache(COMPRESS_CACHE(sbi)); free_node_cache: f2fs_destroy_cache(NODE_CACHE(sbi)); free_meta_cache: -- 2.49.0 ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [f2fs-dev] [PATCH v1 08/12] f2fs: cache: initialize compress cache @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-kernel, linux-f2fs-devel This patch introduces compress_blocks in f2fs_sb_info structure, initializes and destroys the compress cache during filesystem mount and unmount. It adds an ino union field in struct f2fs_cached_block (sharing space with writeback linkage for zero memory overhead) to track per-inode cached blocks, and registers compress cache into the memory shrinker. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 9 ++++++++- fs/f2fs/cache.h | 5 ++++- fs/f2fs/f2fs.h | 6 ++++++ fs/f2fs/super.c | 9 ++++++++- 4 files changed, 26 insertions(+), 3 deletions(-) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index 3176d62ce25b..2cecf6287207 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -194,7 +194,10 @@ static struct f2fs_cached_block *f2fs_create_cache( entry->index = index; atomic_set(&entry->refcount, 0); - entry->next_entry = NULL; + if (!IS_COMPRESS_CACHE(cache)) + entry->next_entry = NULL; + else + entry->ino = 0; INIT_LIST_HEAD(&entry->list); entry->cache = cache; @@ -616,6 +619,10 @@ unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi, return freed; freed += f2fs_do_shrink_cache(NODE_CACHE(sbi), nr_to_scan - freed); + if (freed >= nr_to_scan) + return freed; + + freed += f2fs_do_shrink_cache(COMPRESS_CACHE(sbi), nr_to_scan - freed); return freed; } diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index 397019cfb861..23583a52b8f8 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -38,6 +38,7 @@ struct f2fs_sb_info; enum f2fs_cache_type { F2FS_META_CACHE, F2FS_NODE_CACHE, + F2FS_COMPRESS_CACHE, }; /* Main cache control structure (per sb_info) */ @@ -47,12 +48,13 @@ struct f2fs_cached_block_list { spinlock_t tree_lock; /* Lock for radix tree */ struct list_head lru_list; /* Single global LRU list */ spinlock_t list_lock; /* Lock for LRU list */ - enum f2fs_cache_type type; /* Cache type (Node or Meta) */ + enum f2fs_cache_type type; /* Cache type (Node, Meta, Compress) */ unsigned long num_entries; /* Current number of entries */ }; #define IS_META_CACHE(cache) (cache->type == F2FS_META_CACHE) #define IS_NODE_CACHE(cache) (cache->type == F2FS_NODE_CACHE) +#define IS_COMPRESS_CACHE(cache) (cache->type == F2FS_COMPRESS_CACHE) /* Flags for f2fs_cached_block state */ enum f2fs_cached_state { @@ -182,6 +184,7 @@ void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi); #define META_CACHE(sbi) (&(sbi)->meta_blocks) #define NODE_CACHE(sbi) (&(sbi)->node_blocks) +#define COMPRESS_CACHE(sbi) (&(sbi)->compress_blocks) #define f2fs_find_meta_cache(sbi, blkaddr) \ f2fs_find_cache(META_CACHE(sbi), blkaddr) diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 4dd165ac05c9..36a6afbf3ba1 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -2104,6 +2104,7 @@ struct f2fs_sb_info { /* f2fs internal cache */ struct f2fs_cached_block_list meta_blocks; struct f2fs_cached_block_list node_blocks; + struct f2fs_cached_block_list compress_blocks; /* internal cache flush thread */ struct f2fs_cache_kthread cache_thread; @@ -2325,6 +2326,11 @@ static inline bool f2fs_is_node_cache(struct f2fs_cached_block *entry) return entry->cache && entry->cache == NODE_CACHE(entry->cache->sbi); } +static inline bool f2fs_is_compress_cache(struct f2fs_cached_block *entry) +{ + return entry->cache && entry->cache == COMPRESS_CACHE(entry->cache->sbi); +} + static inline struct f2fs_bio *F2FS_BIO(struct bio *bio) { return container_of(bio, struct f2fs_bio, bio); diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index bc1cff6cad0d..ac3a94736a91 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -2030,6 +2030,7 @@ static void f2fs_put_super(struct super_block *sb) f2fs_destroy_compress_inode(sbi); + f2fs_destroy_cache(COMPRESS_CACHE(sbi)); f2fs_destroy_cache(NODE_CACHE(sbi)); f2fs_destroy_cache(META_CACHE(sbi)); @@ -5203,10 +5204,14 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) if (err) goto free_meta_cache; + err = f2fs_init_cache(sbi, COMPRESS_CACHE(sbi), F2FS_COMPRESS_CACHE); + if (err) + goto free_node_cache; + err = f2fs_get_valid_checkpoint(sbi); if (err) { f2fs_err(sbi, "Failed to get valid F2FS checkpoint"); - goto free_node_cache; + goto free_compress_cache; } if (__is_set_ckpt_flags(F2FS_CKPT(sbi), CP_QUOTA_NEED_FSCK_FLAG)) @@ -5519,6 +5524,8 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) free_devices: destroy_device_list(sbi); kvfree(sbi->ckpt); +free_compress_cache: + f2fs_destroy_cache(COMPRESS_CACHE(sbi)); free_node_cache: f2fs_destroy_cache(NODE_CACHE(sbi)); free_meta_cache: -- 2.49.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [PATCH v1 09/12] f2fs: cache: use compress cache 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel -1 siblings, 0 replies; 34+ messages in thread From: Chao Yu @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-f2fs-devel, linux-kernel, Chao Yu This patch migrates compressed cluster caching from the fake VFS inode page cache (sbi->compress_inode) to compress cache (sbi->compress_blocks). It converts compression caching and decompression paths to use compress cache APIs, uses entry->ino for per-inode invalidation, and removes sbi->compress_inode. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 2 +- fs/f2fs/cache.h | 1 + fs/f2fs/compress.c | 149 ++++++++++++++++------------------------ fs/f2fs/debug.c | 9 +-- fs/f2fs/f2fs.h | 8 +-- fs/f2fs/inode.c | 49 ++----------- fs/f2fs/node.c | 2 +- fs/f2fs/super.c | 15 ++-- include/linux/f2fs_fs.h | 1 - 9 files changed, 79 insertions(+), 157 deletions(-) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index 2cecf6287207..173bbf3aa94c 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -414,7 +414,7 @@ static void f2fs_do_truncate_cache(struct f2fs_cached_block *entry, spin_unlock(&cache->list_lock); } -static void f2fs_truncate_cache(struct f2fs_cached_block *entry, +void f2fs_truncate_cache(struct f2fs_cached_block *entry, bool drop_dirty) { f2fs_lock_cache(entry); diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index 23583a52b8f8..12c061d28d23 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -169,6 +169,7 @@ unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache, unsigned int max_items, int tag); void f2fs_cache_gang_release(struct f2fs_cached_block **entries, unsigned int nr_entries); +void f2fs_truncate_cache(struct f2fs_cached_block *entry, bool drop_dirty); int f2fs_writeback_cache(struct f2fs_cached_block_list *cache, bool sync); void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache); void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c index 676a5559357f..2eb7614d3610 100644 --- a/fs/f2fs/compress.c +++ b/fs/f2fs/compress.c @@ -1918,30 +1918,19 @@ unsigned int f2fs_cluster_blocks_are_contiguous(struct dnode_of_data *dn, return compressed ? i - 1 : i; } -const struct address_space_operations f2fs_compress_aops = { - .release_folio = f2fs_release_folio, - .invalidate_folio = f2fs_invalidate_folio, - .migrate_folio = filemap_migrate_folio, -}; - -struct address_space *COMPRESS_MAPPING(struct f2fs_sb_info *sbi) -{ - return sbi->compress_inode->i_mapping; -} - void f2fs_invalidate_compress_pages_range(struct f2fs_sb_info *sbi, block_t blkaddr, unsigned int len) { - if (!sbi->compress_inode) + if (!test_opt(sbi, COMPRESS_CACHE)) return; - invalidate_mapping_pages(COMPRESS_MAPPING(sbi), blkaddr, blkaddr + len - 1); + + f2fs_drop_cache_range(COMPRESS_CACHE(sbi), blkaddr, len, false); } static void f2fs_cache_compressed_page(struct f2fs_sb_info *sbi, struct folio *folio, nid_t ino, block_t blkaddr) { - struct folio *cfolio; - int ret; + struct f2fs_cached_block *entry; if (!test_opt(sbi, COMPRESS_CACHE)) return; @@ -1952,49 +1941,43 @@ static void f2fs_cache_compressed_page(struct f2fs_sb_info *sbi, if (!f2fs_available_free_memory(sbi, COMPRESS_PAGE)) return; - cfolio = filemap_get_folio(COMPRESS_MAPPING(sbi), blkaddr); - if (!IS_ERR(cfolio)) { - f2fs_folio_put(cfolio, false); + entry = f2fs_find_cache(COMPRESS_CACHE(sbi), blkaddr); + if (!IS_ERR(entry)) { + f2fs_put_cache(entry, false); return; } - cfolio = filemap_alloc_folio(__GFP_NOWARN | __GFP_IO, 0, NULL); - if (!cfolio) - return; - - ret = filemap_add_folio(COMPRESS_MAPPING(sbi), cfolio, - blkaddr, GFP_NOFS); - if (ret) { - f2fs_folio_put(cfolio, false); + entry = f2fs_grab_cache(COMPRESS_CACHE(sbi), blkaddr, + F2FS_CACHE_LOCK_CREATE); + if (IS_ERR(entry)) return; - } - - folio_set_f2fs_data(cfolio, ino); - memcpy(folio_address(cfolio), folio_address(folio), PAGE_SIZE); - folio_mark_uptodate(cfolio); - f2fs_folio_put(cfolio, true); + entry->ino = ino; + memcpy(cache_address(entry), folio_address(folio), PAGE_SIZE); + f2fs_cache_set_uptodate(entry); + f2fs_put_cache(entry, true); } bool f2fs_load_compressed_folio(struct f2fs_sb_info *sbi, struct folio *folio, block_t blkaddr) { - struct folio *cfolio; + struct f2fs_cached_block *entry; bool hitted = false; if (!test_opt(sbi, COMPRESS_CACHE)) return false; - cfolio = f2fs_filemap_get_folio(COMPRESS_MAPPING(sbi), - blkaddr, FGP_LOCK | FGP_NOWAIT, GFP_NOFS); - if (!IS_ERR(cfolio)) { - if (folio_test_uptodate(cfolio)) { + entry = f2fs_find_cache(COMPRESS_CACHE(sbi), blkaddr); + if (!IS_ERR(entry)) { + f2fs_lock_cache(entry); + if (f2fs_is_compress_cache(entry) && + f2fs_cache_test_uptodate(entry)) { atomic_inc(&sbi->compress_page_hit); memcpy(folio_address(folio), - folio_address(cfolio), folio_size(folio)); + cache_address(entry), folio_size(folio)); hitted = true; } - f2fs_folio_put(cfolio, true); + f2fs_put_cache(entry, true); } return hitted; @@ -2002,71 +1985,59 @@ bool f2fs_load_compressed_folio(struct f2fs_sb_info *sbi, struct folio *folio, void f2fs_invalidate_compress_pages(struct f2fs_sb_info *sbi, nid_t ino) { - struct address_space *mapping = COMPRESS_MAPPING(sbi); - struct folio_batch fbatch; - pgoff_t index = 0; - pgoff_t end = MAX_BLKADDR(sbi); + struct f2fs_cached_block_list *cache = COMPRESS_CACHE(sbi); + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; + unsigned long flags; + pgoff_t index = 0, end = ULONG_MAX; + int nr; + int i; - if (!mapping->nrpages) + if (!test_opt(sbi, COMPRESS_CACHE)) return; - - folio_batch_init(&fbatch); - - do { - unsigned int nr, i; - - nr = filemap_get_folios(mapping, &index, end - 1, &fbatch); - if (!nr) +next: + spin_lock_irqsave(&cache->tree_lock, flags); + nr = radix_tree_gang_lookup(&cache->root, (void **)entries, index, + min((unsigned long)F2FS_ONSTACK_CACHES, end - index)); + if (!nr) + goto out_unlock; + + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; + + if (entry->index >= end) { + nr = i; break; + } + f2fs_cache_get(entry); + } +out_unlock: + spin_unlock_irqrestore(&cache->tree_lock, flags); + if (!nr) + return; + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; - for (i = 0; i < nr; i++) { - struct folio *folio = fbatch.folios[i]; + index = entry->index + 1; - folio_lock(folio); - if (folio->mapping != mapping) { - folio_unlock(folio); - continue; - } + if (IS_COMPRESS_CACHE(cache) && entry->ino != ino) + continue; - if (ino != folio_get_f2fs_data(folio)) { - folio_unlock(folio); - continue; - } + f2fs_truncate_cache(entry, false); + } + f2fs_cache_gang_release(entries, nr); - generic_error_remove_folio(mapping, folio); - folio_unlock(folio); - } - folio_batch_release(&fbatch); - cond_resched(); - } while (index < end); + if (index < end) + goto next; } -int f2fs_init_compress_inode(struct f2fs_sb_info *sbi) +void f2fs_init_compress_cache_context(struct f2fs_sb_info *sbi) { - struct inode *inode; - if (!test_opt(sbi, COMPRESS_CACHE)) - return 0; - - inode = f2fs_iget(sbi->sb, F2FS_COMPRESS_INO(sbi)); - if (IS_ERR(inode)) - return PTR_ERR(inode); - sbi->compress_inode = inode; + return; sbi->compress_percent = COMPRESS_PERCENT; sbi->compress_watermark = COMPRESS_WATERMARK; - atomic_set(&sbi->compress_page_hit, 0); - - return 0; -} - -void f2fs_destroy_compress_inode(struct f2fs_sb_info *sbi) -{ - if (!sbi->compress_inode) - return; - iput(sbi->compress_inode); - sbi->compress_inode = NULL; } int f2fs_init_page_array_cache(struct f2fs_sb_info *sbi) diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c index bedaade92677..8cd06f7ba9e7 100644 --- a/fs/f2fs/debug.c +++ b/fs/f2fs/debug.c @@ -225,8 +225,8 @@ static void update_general_status(struct f2fs_sb_info *sbi) si->node_caches = NODE_CACHE(sbi)->num_entries; si->meta_caches = META_CACHE(sbi)->num_entries; #ifdef CONFIG_F2FS_FS_COMPRESSION - if (sbi->compress_inode) { - si->compress_pages = COMPRESS_MAPPING(sbi)->nrpages; + if (test_opt(sbi, COMPRESS_CACHE)) { + si->compress_pages = COMPRESS_CACHE(sbi)->num_entries; si->compress_page_hit = atomic_read(&sbi->compress_page_hit); } #endif @@ -386,10 +386,11 @@ static void update_mem_info(struct f2fs_sb_info *sbi) si->page_mem += (unsigned long long)NODE_CACHE(sbi)->num_entries << PAGE_SHIFT; si->cache_mem += NODE_CACHE(sbi)->num_entries * sizeof(struct f2fs_cached_block); #ifdef CONFIG_F2FS_FS_COMPRESSION - if (sbi->compress_inode) { - unsigned long npages = COMPRESS_MAPPING(sbi)->nrpages; + if (test_opt(sbi, COMPRESS_CACHE)) { + unsigned long npages = COMPRESS_CACHE(sbi)->num_entries; si->page_mem += (unsigned long long)npages << PAGE_SHIFT; + si->cache_mem += npages * sizeof(struct f2fs_cached_block); } #endif } diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 36a6afbf3ba1..0a749a59f386 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -2075,7 +2075,6 @@ struct f2fs_sb_info { u32 compr_new_inode; /* For compressed block cache */ - struct inode *compress_inode; /* cache compressed blocks */ unsigned int compress_percent; /* cache page percentage */ unsigned int compress_watermark; /* cache page watermark */ atomic_t compress_page_hit; /* cache hit count */ @@ -4817,13 +4816,11 @@ unsigned int f2fs_cluster_blocks_are_contiguous(struct dnode_of_data *dn, int f2fs_init_compress_ctx(struct compress_ctx *cc); void f2fs_destroy_compress_ctx(struct compress_ctx *cc, bool reuse); void f2fs_init_compress_info(struct f2fs_sb_info *sbi); -int f2fs_init_compress_inode(struct f2fs_sb_info *sbi); -void f2fs_destroy_compress_inode(struct f2fs_sb_info *sbi); +void f2fs_init_compress_cache_context(struct f2fs_sb_info *sbi); int f2fs_init_page_array_cache(struct f2fs_sb_info *sbi); void f2fs_destroy_page_array_cache(struct f2fs_sb_info *sbi); int __init f2fs_init_compress_cache(void); void f2fs_destroy_compress_cache(void); -struct address_space *COMPRESS_MAPPING(struct f2fs_sb_info *sbi); void f2fs_invalidate_compress_pages_range(struct f2fs_sb_info *sbi, block_t blkaddr, unsigned int len); bool f2fs_load_compressed_folio(struct f2fs_sb_info *sbi, struct folio *folio, @@ -4872,8 +4869,7 @@ static inline void f2fs_put_folio_dic(struct folio *folio, bool in_task) static inline unsigned int f2fs_cluster_blocks_are_contiguous( struct dnode_of_data *dn, unsigned int ofs_in_node) { return 0; } static inline bool f2fs_sanity_check_cluster(struct dnode_of_data *dn) { return false; } -static inline int f2fs_init_compress_inode(struct f2fs_sb_info *sbi) { return 0; } -static inline void f2fs_destroy_compress_inode(struct f2fs_sb_info *sbi) { } +static inline void f2fs_init_compress_cache_context(struct f2fs_sb_info *sbi) { } static inline int f2fs_init_page_array_cache(struct f2fs_sb_info *sbi) { return 0; } static inline void f2fs_destroy_page_array_cache(struct f2fs_sb_info *sbi) { } static inline int __init f2fs_init_compress_cache(void) { return 0; } diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c index 421788a9ea24..579da36b600d 100644 --- a/fs/f2fs/inode.c +++ b/fs/f2fs/inode.c @@ -573,15 +573,6 @@ static int do_read_inode(struct inode *inode) return 0; } -static bool is_meta_ino(struct f2fs_sb_info *sbi, unsigned int ino) -{ -#ifdef CONFIG_F2FS_FS_COMPRESSION - if (test_opt(sbi, COMPRESS_CACHE) && ino == F2FS_COMPRESS_INO(sbi)) - return true; -#endif - return false; -} - struct inode *f2fs_iget(struct super_block *sb, unsigned long ino) { struct f2fs_sb_info *sbi = F2FS_SB(sb); @@ -593,42 +584,17 @@ struct inode *f2fs_iget(struct super_block *sb, unsigned long ino) return ERR_PTR(-ENOMEM); if (!(inode_state_read_once(inode) & I_NEW)) { - if (is_meta_ino(sbi, ino)) { - f2fs_err(sbi, "inaccessible inode: %lu, run fsck to repair", ino); - set_sbi_flag(sbi, SBI_NEED_FSCK); - ret = -EFSCORRUPTED; - trace_f2fs_iget_exit(inode, ret); - iput(inode); - f2fs_handle_error(sbi, ERROR_CORRUPTED_INODE); - fserror_report_file_metadata(inode, ret, GFP_NOFS); - return ERR_PTR(ret); - } - trace_f2fs_iget(inode); return inode; } - if (is_meta_ino(sbi, ino)) - goto make_now; - ret = do_read_inode(inode); if (ret) goto bad_inode; -make_now: + f2fs_set_inode_flags(inode); - if (ino == F2FS_COMPRESS_INO(sbi)) { -#ifdef CONFIG_F2FS_FS_COMPRESSION - inode->i_mapping->a_ops = &f2fs_compress_aops; - /* - * generic_error_remove_folio only truncates pages of regular - * inode - */ - inode->i_mode |= S_IFREG; -#endif - mapping_set_gfp_mask(inode->i_mapping, - GFP_NOFS | __GFP_HIGHMEM | __GFP_MOVABLE); - } else if (S_ISREG(inode->i_mode)) { + if (S_ISREG(inode->i_mode)) { inode->i_op = &f2fs_file_inode_operations; inode->i_fop = &f2fs_file_operations; inode->i_mapping->a_ops = &f2fs_dblock_aops; @@ -879,7 +845,7 @@ static void f2fs_evict_inode_work(struct work_struct *work) /* * Return true, if we shouldn't go through post_evict_inode. */ -static bool f2fs_pre_evict_inode(struct inode *inode) +static void f2fs_pre_evict_inode(struct inode *inode) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct f2fs_inode_info *fi = F2FS_I(inode); @@ -905,17 +871,12 @@ static bool f2fs_pre_evict_inode(struct inode *inode) test_opt(sbi, COMPRESS_CACHE) && f2fs_compressed_file(inode)) f2fs_invalidate_compress_pages(sbi, inode->i_ino); - if (inode->i_ino == F2FS_COMPRESS_INO(sbi)) - return true; - f2fs_bug_on(sbi, get_dirty_pages(inode)); f2fs_remove_dirty_inode(inode); f2fs_remove_donate_inode(inode); if (!IS_DEVICE_ALIASING(inode)) f2fs_destroy_extent_tree(inode); - - return false; } static void f2fs_delete_inode(struct inode *inode) @@ -1077,15 +1038,13 @@ static void f2fs_post_evict_inode(struct inode *inode) */ void f2fs_evict_inode(struct inode *inode) { - if (f2fs_pre_evict_inode(inode)) - goto clear_out; + f2fs_pre_evict_inode(inode); if (!inode->i_nlink && !is_bad_inode(inode)) f2fs_delete_inode(inode); f2fs_post_evict_inode(inode); -clear_out: fscrypt_put_encryption_info(inode); clear_inode(inode); } diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c index 9482c86b5506..4b611ed3d8ef 100644 --- a/fs/f2fs/node.c +++ b/fs/f2fs/node.c @@ -119,7 +119,7 @@ bool f2fs_available_free_memory(struct f2fs_sb_info *sbi, int type) * exceed threshold, deny caching compress page. */ res = (free_ram > avail_ram * sbi->compress_watermark / 100) && - (COMPRESS_MAPPING(sbi)->nrpages < + (COMPRESS_CACHE(sbi)->num_entries < free_ram * sbi->compress_percent / 100); #else res = false; diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index ac3a94736a91..64c39286486d 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -2028,8 +2028,6 @@ static void f2fs_put_super(struct super_block *sb) f2fs_bug_on(sbi, sbi->fsync_node_num); - f2fs_destroy_compress_inode(sbi); - f2fs_destroy_cache(COMPRESS_CACHE(sbi)); f2fs_destroy_cache(NODE_CACHE(sbi)); f2fs_destroy_cache(META_CACHE(sbi)); @@ -5257,6 +5255,8 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) f2fs_init_fsync_node_info(sbi); + f2fs_init_compress_cache_context(sbi); + /* setup checkpoint request control and start checkpoint issue thread */ f2fs_init_ckpt_req_control(sbi); if (!f2fs_readonly(sb) && !test_opt(sbi, DISABLE_CHECKPOINT) && @@ -5327,13 +5327,9 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) goto free_ino_entry; } - err = f2fs_init_compress_inode(sbi); - if (err) - goto free_root_inode; - err = f2fs_register_sysfs(sbi); if (err) - goto free_compress_inode; + goto free_root_inode; sbi->umount_lock_holder = current; #ifdef CONFIG_QUOTA @@ -5501,8 +5497,6 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) /* evict some inodes being cached by GC */ evict_inodes(sb); f2fs_unregister_sysfs(sbi); -free_compress_inode: - f2fs_destroy_compress_inode(sbi); free_root_inode: dput(sb->s_root); sb->s_root = NULL; @@ -5619,7 +5613,8 @@ static void kill_f2fs_super(struct super_block *sb) * compress inode cache. */ if (test_opt(sbi, COMPRESS_CACHE)) - truncate_inode_pages_final(COMPRESS_MAPPING(sbi)); + f2fs_invalidate_compress_pages_range(sbi, + 0, UINT_MAX); #endif if (is_sbi_flag_set(sbi, SBI_IS_DIRTY) || diff --git a/include/linux/f2fs_fs.h b/include/linux/f2fs_fs.h index 105cfeedea74..53344e1f2b44 100644 --- a/include/linux/f2fs_fs.h +++ b/include/linux/f2fs_fs.h @@ -34,7 +34,6 @@ #define F2FS_RESERVED_NODE_NUM 3 #define F2FS_ROOT_INO(sbi) ((sbi)->root_ino_num) -#define F2FS_COMPRESS_INO(sbi) (NM_I(sbi)->max_nid) #define F2FS_MAX_QUOTAS 3 -- 2.49.0 ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [f2fs-dev] [PATCH v1 09/12] f2fs: cache: use compress cache @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-kernel, linux-f2fs-devel This patch migrates compressed cluster caching from the fake VFS inode page cache (sbi->compress_inode) to compress cache (sbi->compress_blocks). It converts compression caching and decompression paths to use compress cache APIs, uses entry->ino for per-inode invalidation, and removes sbi->compress_inode. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 2 +- fs/f2fs/cache.h | 1 + fs/f2fs/compress.c | 149 ++++++++++++++++------------------------ fs/f2fs/debug.c | 9 +-- fs/f2fs/f2fs.h | 8 +-- fs/f2fs/inode.c | 49 ++----------- fs/f2fs/node.c | 2 +- fs/f2fs/super.c | 15 ++-- include/linux/f2fs_fs.h | 1 - 9 files changed, 79 insertions(+), 157 deletions(-) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index 2cecf6287207..173bbf3aa94c 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -414,7 +414,7 @@ static void f2fs_do_truncate_cache(struct f2fs_cached_block *entry, spin_unlock(&cache->list_lock); } -static void f2fs_truncate_cache(struct f2fs_cached_block *entry, +void f2fs_truncate_cache(struct f2fs_cached_block *entry, bool drop_dirty) { f2fs_lock_cache(entry); diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index 23583a52b8f8..12c061d28d23 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -169,6 +169,7 @@ unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache, unsigned int max_items, int tag); void f2fs_cache_gang_release(struct f2fs_cached_block **entries, unsigned int nr_entries); +void f2fs_truncate_cache(struct f2fs_cached_block *entry, bool drop_dirty); int f2fs_writeback_cache(struct f2fs_cached_block_list *cache, bool sync); void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache); void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c index 676a5559357f..2eb7614d3610 100644 --- a/fs/f2fs/compress.c +++ b/fs/f2fs/compress.c @@ -1918,30 +1918,19 @@ unsigned int f2fs_cluster_blocks_are_contiguous(struct dnode_of_data *dn, return compressed ? i - 1 : i; } -const struct address_space_operations f2fs_compress_aops = { - .release_folio = f2fs_release_folio, - .invalidate_folio = f2fs_invalidate_folio, - .migrate_folio = filemap_migrate_folio, -}; - -struct address_space *COMPRESS_MAPPING(struct f2fs_sb_info *sbi) -{ - return sbi->compress_inode->i_mapping; -} - void f2fs_invalidate_compress_pages_range(struct f2fs_sb_info *sbi, block_t blkaddr, unsigned int len) { - if (!sbi->compress_inode) + if (!test_opt(sbi, COMPRESS_CACHE)) return; - invalidate_mapping_pages(COMPRESS_MAPPING(sbi), blkaddr, blkaddr + len - 1); + + f2fs_drop_cache_range(COMPRESS_CACHE(sbi), blkaddr, len, false); } static void f2fs_cache_compressed_page(struct f2fs_sb_info *sbi, struct folio *folio, nid_t ino, block_t blkaddr) { - struct folio *cfolio; - int ret; + struct f2fs_cached_block *entry; if (!test_opt(sbi, COMPRESS_CACHE)) return; @@ -1952,49 +1941,43 @@ static void f2fs_cache_compressed_page(struct f2fs_sb_info *sbi, if (!f2fs_available_free_memory(sbi, COMPRESS_PAGE)) return; - cfolio = filemap_get_folio(COMPRESS_MAPPING(sbi), blkaddr); - if (!IS_ERR(cfolio)) { - f2fs_folio_put(cfolio, false); + entry = f2fs_find_cache(COMPRESS_CACHE(sbi), blkaddr); + if (!IS_ERR(entry)) { + f2fs_put_cache(entry, false); return; } - cfolio = filemap_alloc_folio(__GFP_NOWARN | __GFP_IO, 0, NULL); - if (!cfolio) - return; - - ret = filemap_add_folio(COMPRESS_MAPPING(sbi), cfolio, - blkaddr, GFP_NOFS); - if (ret) { - f2fs_folio_put(cfolio, false); + entry = f2fs_grab_cache(COMPRESS_CACHE(sbi), blkaddr, + F2FS_CACHE_LOCK_CREATE); + if (IS_ERR(entry)) return; - } - - folio_set_f2fs_data(cfolio, ino); - memcpy(folio_address(cfolio), folio_address(folio), PAGE_SIZE); - folio_mark_uptodate(cfolio); - f2fs_folio_put(cfolio, true); + entry->ino = ino; + memcpy(cache_address(entry), folio_address(folio), PAGE_SIZE); + f2fs_cache_set_uptodate(entry); + f2fs_put_cache(entry, true); } bool f2fs_load_compressed_folio(struct f2fs_sb_info *sbi, struct folio *folio, block_t blkaddr) { - struct folio *cfolio; + struct f2fs_cached_block *entry; bool hitted = false; if (!test_opt(sbi, COMPRESS_CACHE)) return false; - cfolio = f2fs_filemap_get_folio(COMPRESS_MAPPING(sbi), - blkaddr, FGP_LOCK | FGP_NOWAIT, GFP_NOFS); - if (!IS_ERR(cfolio)) { - if (folio_test_uptodate(cfolio)) { + entry = f2fs_find_cache(COMPRESS_CACHE(sbi), blkaddr); + if (!IS_ERR(entry)) { + f2fs_lock_cache(entry); + if (f2fs_is_compress_cache(entry) && + f2fs_cache_test_uptodate(entry)) { atomic_inc(&sbi->compress_page_hit); memcpy(folio_address(folio), - folio_address(cfolio), folio_size(folio)); + cache_address(entry), folio_size(folio)); hitted = true; } - f2fs_folio_put(cfolio, true); + f2fs_put_cache(entry, true); } return hitted; @@ -2002,71 +1985,59 @@ bool f2fs_load_compressed_folio(struct f2fs_sb_info *sbi, struct folio *folio, void f2fs_invalidate_compress_pages(struct f2fs_sb_info *sbi, nid_t ino) { - struct address_space *mapping = COMPRESS_MAPPING(sbi); - struct folio_batch fbatch; - pgoff_t index = 0; - pgoff_t end = MAX_BLKADDR(sbi); + struct f2fs_cached_block_list *cache = COMPRESS_CACHE(sbi); + struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES]; + unsigned long flags; + pgoff_t index = 0, end = ULONG_MAX; + int nr; + int i; - if (!mapping->nrpages) + if (!test_opt(sbi, COMPRESS_CACHE)) return; - - folio_batch_init(&fbatch); - - do { - unsigned int nr, i; - - nr = filemap_get_folios(mapping, &index, end - 1, &fbatch); - if (!nr) +next: + spin_lock_irqsave(&cache->tree_lock, flags); + nr = radix_tree_gang_lookup(&cache->root, (void **)entries, index, + min((unsigned long)F2FS_ONSTACK_CACHES, end - index)); + if (!nr) + goto out_unlock; + + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; + + if (entry->index >= end) { + nr = i; break; + } + f2fs_cache_get(entry); + } +out_unlock: + spin_unlock_irqrestore(&cache->tree_lock, flags); + if (!nr) + return; + for (i = 0; i < nr; i++) { + struct f2fs_cached_block *entry = entries[i]; - for (i = 0; i < nr; i++) { - struct folio *folio = fbatch.folios[i]; + index = entry->index + 1; - folio_lock(folio); - if (folio->mapping != mapping) { - folio_unlock(folio); - continue; - } + if (IS_COMPRESS_CACHE(cache) && entry->ino != ino) + continue; - if (ino != folio_get_f2fs_data(folio)) { - folio_unlock(folio); - continue; - } + f2fs_truncate_cache(entry, false); + } + f2fs_cache_gang_release(entries, nr); - generic_error_remove_folio(mapping, folio); - folio_unlock(folio); - } - folio_batch_release(&fbatch); - cond_resched(); - } while (index < end); + if (index < end) + goto next; } -int f2fs_init_compress_inode(struct f2fs_sb_info *sbi) +void f2fs_init_compress_cache_context(struct f2fs_sb_info *sbi) { - struct inode *inode; - if (!test_opt(sbi, COMPRESS_CACHE)) - return 0; - - inode = f2fs_iget(sbi->sb, F2FS_COMPRESS_INO(sbi)); - if (IS_ERR(inode)) - return PTR_ERR(inode); - sbi->compress_inode = inode; + return; sbi->compress_percent = COMPRESS_PERCENT; sbi->compress_watermark = COMPRESS_WATERMARK; - atomic_set(&sbi->compress_page_hit, 0); - - return 0; -} - -void f2fs_destroy_compress_inode(struct f2fs_sb_info *sbi) -{ - if (!sbi->compress_inode) - return; - iput(sbi->compress_inode); - sbi->compress_inode = NULL; } int f2fs_init_page_array_cache(struct f2fs_sb_info *sbi) diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c index bedaade92677..8cd06f7ba9e7 100644 --- a/fs/f2fs/debug.c +++ b/fs/f2fs/debug.c @@ -225,8 +225,8 @@ static void update_general_status(struct f2fs_sb_info *sbi) si->node_caches = NODE_CACHE(sbi)->num_entries; si->meta_caches = META_CACHE(sbi)->num_entries; #ifdef CONFIG_F2FS_FS_COMPRESSION - if (sbi->compress_inode) { - si->compress_pages = COMPRESS_MAPPING(sbi)->nrpages; + if (test_opt(sbi, COMPRESS_CACHE)) { + si->compress_pages = COMPRESS_CACHE(sbi)->num_entries; si->compress_page_hit = atomic_read(&sbi->compress_page_hit); } #endif @@ -386,10 +386,11 @@ static void update_mem_info(struct f2fs_sb_info *sbi) si->page_mem += (unsigned long long)NODE_CACHE(sbi)->num_entries << PAGE_SHIFT; si->cache_mem += NODE_CACHE(sbi)->num_entries * sizeof(struct f2fs_cached_block); #ifdef CONFIG_F2FS_FS_COMPRESSION - if (sbi->compress_inode) { - unsigned long npages = COMPRESS_MAPPING(sbi)->nrpages; + if (test_opt(sbi, COMPRESS_CACHE)) { + unsigned long npages = COMPRESS_CACHE(sbi)->num_entries; si->page_mem += (unsigned long long)npages << PAGE_SHIFT; + si->cache_mem += npages * sizeof(struct f2fs_cached_block); } #endif } diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 36a6afbf3ba1..0a749a59f386 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -2075,7 +2075,6 @@ struct f2fs_sb_info { u32 compr_new_inode; /* For compressed block cache */ - struct inode *compress_inode; /* cache compressed blocks */ unsigned int compress_percent; /* cache page percentage */ unsigned int compress_watermark; /* cache page watermark */ atomic_t compress_page_hit; /* cache hit count */ @@ -4817,13 +4816,11 @@ unsigned int f2fs_cluster_blocks_are_contiguous(struct dnode_of_data *dn, int f2fs_init_compress_ctx(struct compress_ctx *cc); void f2fs_destroy_compress_ctx(struct compress_ctx *cc, bool reuse); void f2fs_init_compress_info(struct f2fs_sb_info *sbi); -int f2fs_init_compress_inode(struct f2fs_sb_info *sbi); -void f2fs_destroy_compress_inode(struct f2fs_sb_info *sbi); +void f2fs_init_compress_cache_context(struct f2fs_sb_info *sbi); int f2fs_init_page_array_cache(struct f2fs_sb_info *sbi); void f2fs_destroy_page_array_cache(struct f2fs_sb_info *sbi); int __init f2fs_init_compress_cache(void); void f2fs_destroy_compress_cache(void); -struct address_space *COMPRESS_MAPPING(struct f2fs_sb_info *sbi); void f2fs_invalidate_compress_pages_range(struct f2fs_sb_info *sbi, block_t blkaddr, unsigned int len); bool f2fs_load_compressed_folio(struct f2fs_sb_info *sbi, struct folio *folio, @@ -4872,8 +4869,7 @@ static inline void f2fs_put_folio_dic(struct folio *folio, bool in_task) static inline unsigned int f2fs_cluster_blocks_are_contiguous( struct dnode_of_data *dn, unsigned int ofs_in_node) { return 0; } static inline bool f2fs_sanity_check_cluster(struct dnode_of_data *dn) { return false; } -static inline int f2fs_init_compress_inode(struct f2fs_sb_info *sbi) { return 0; } -static inline void f2fs_destroy_compress_inode(struct f2fs_sb_info *sbi) { } +static inline void f2fs_init_compress_cache_context(struct f2fs_sb_info *sbi) { } static inline int f2fs_init_page_array_cache(struct f2fs_sb_info *sbi) { return 0; } static inline void f2fs_destroy_page_array_cache(struct f2fs_sb_info *sbi) { } static inline int __init f2fs_init_compress_cache(void) { return 0; } diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c index 421788a9ea24..579da36b600d 100644 --- a/fs/f2fs/inode.c +++ b/fs/f2fs/inode.c @@ -573,15 +573,6 @@ static int do_read_inode(struct inode *inode) return 0; } -static bool is_meta_ino(struct f2fs_sb_info *sbi, unsigned int ino) -{ -#ifdef CONFIG_F2FS_FS_COMPRESSION - if (test_opt(sbi, COMPRESS_CACHE) && ino == F2FS_COMPRESS_INO(sbi)) - return true; -#endif - return false; -} - struct inode *f2fs_iget(struct super_block *sb, unsigned long ino) { struct f2fs_sb_info *sbi = F2FS_SB(sb); @@ -593,42 +584,17 @@ struct inode *f2fs_iget(struct super_block *sb, unsigned long ino) return ERR_PTR(-ENOMEM); if (!(inode_state_read_once(inode) & I_NEW)) { - if (is_meta_ino(sbi, ino)) { - f2fs_err(sbi, "inaccessible inode: %lu, run fsck to repair", ino); - set_sbi_flag(sbi, SBI_NEED_FSCK); - ret = -EFSCORRUPTED; - trace_f2fs_iget_exit(inode, ret); - iput(inode); - f2fs_handle_error(sbi, ERROR_CORRUPTED_INODE); - fserror_report_file_metadata(inode, ret, GFP_NOFS); - return ERR_PTR(ret); - } - trace_f2fs_iget(inode); return inode; } - if (is_meta_ino(sbi, ino)) - goto make_now; - ret = do_read_inode(inode); if (ret) goto bad_inode; -make_now: + f2fs_set_inode_flags(inode); - if (ino == F2FS_COMPRESS_INO(sbi)) { -#ifdef CONFIG_F2FS_FS_COMPRESSION - inode->i_mapping->a_ops = &f2fs_compress_aops; - /* - * generic_error_remove_folio only truncates pages of regular - * inode - */ - inode->i_mode |= S_IFREG; -#endif - mapping_set_gfp_mask(inode->i_mapping, - GFP_NOFS | __GFP_HIGHMEM | __GFP_MOVABLE); - } else if (S_ISREG(inode->i_mode)) { + if (S_ISREG(inode->i_mode)) { inode->i_op = &f2fs_file_inode_operations; inode->i_fop = &f2fs_file_operations; inode->i_mapping->a_ops = &f2fs_dblock_aops; @@ -879,7 +845,7 @@ static void f2fs_evict_inode_work(struct work_struct *work) /* * Return true, if we shouldn't go through post_evict_inode. */ -static bool f2fs_pre_evict_inode(struct inode *inode) +static void f2fs_pre_evict_inode(struct inode *inode) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct f2fs_inode_info *fi = F2FS_I(inode); @@ -905,17 +871,12 @@ static bool f2fs_pre_evict_inode(struct inode *inode) test_opt(sbi, COMPRESS_CACHE) && f2fs_compressed_file(inode)) f2fs_invalidate_compress_pages(sbi, inode->i_ino); - if (inode->i_ino == F2FS_COMPRESS_INO(sbi)) - return true; - f2fs_bug_on(sbi, get_dirty_pages(inode)); f2fs_remove_dirty_inode(inode); f2fs_remove_donate_inode(inode); if (!IS_DEVICE_ALIASING(inode)) f2fs_destroy_extent_tree(inode); - - return false; } static void f2fs_delete_inode(struct inode *inode) @@ -1077,15 +1038,13 @@ static void f2fs_post_evict_inode(struct inode *inode) */ void f2fs_evict_inode(struct inode *inode) { - if (f2fs_pre_evict_inode(inode)) - goto clear_out; + f2fs_pre_evict_inode(inode); if (!inode->i_nlink && !is_bad_inode(inode)) f2fs_delete_inode(inode); f2fs_post_evict_inode(inode); -clear_out: fscrypt_put_encryption_info(inode); clear_inode(inode); } diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c index 9482c86b5506..4b611ed3d8ef 100644 --- a/fs/f2fs/node.c +++ b/fs/f2fs/node.c @@ -119,7 +119,7 @@ bool f2fs_available_free_memory(struct f2fs_sb_info *sbi, int type) * exceed threshold, deny caching compress page. */ res = (free_ram > avail_ram * sbi->compress_watermark / 100) && - (COMPRESS_MAPPING(sbi)->nrpages < + (COMPRESS_CACHE(sbi)->num_entries < free_ram * sbi->compress_percent / 100); #else res = false; diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index ac3a94736a91..64c39286486d 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -2028,8 +2028,6 @@ static void f2fs_put_super(struct super_block *sb) f2fs_bug_on(sbi, sbi->fsync_node_num); - f2fs_destroy_compress_inode(sbi); - f2fs_destroy_cache(COMPRESS_CACHE(sbi)); f2fs_destroy_cache(NODE_CACHE(sbi)); f2fs_destroy_cache(META_CACHE(sbi)); @@ -5257,6 +5255,8 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) f2fs_init_fsync_node_info(sbi); + f2fs_init_compress_cache_context(sbi); + /* setup checkpoint request control and start checkpoint issue thread */ f2fs_init_ckpt_req_control(sbi); if (!f2fs_readonly(sb) && !test_opt(sbi, DISABLE_CHECKPOINT) && @@ -5327,13 +5327,9 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) goto free_ino_entry; } - err = f2fs_init_compress_inode(sbi); - if (err) - goto free_root_inode; - err = f2fs_register_sysfs(sbi); if (err) - goto free_compress_inode; + goto free_root_inode; sbi->umount_lock_holder = current; #ifdef CONFIG_QUOTA @@ -5501,8 +5497,6 @@ static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) /* evict some inodes being cached by GC */ evict_inodes(sb); f2fs_unregister_sysfs(sbi); -free_compress_inode: - f2fs_destroy_compress_inode(sbi); free_root_inode: dput(sb->s_root); sb->s_root = NULL; @@ -5619,7 +5613,8 @@ static void kill_f2fs_super(struct super_block *sb) * compress inode cache. */ if (test_opt(sbi, COMPRESS_CACHE)) - truncate_inode_pages_final(COMPRESS_MAPPING(sbi)); + f2fs_invalidate_compress_pages_range(sbi, + 0, UINT_MAX); #endif if (is_sbi_flag_set(sbi, SBI_IS_DIRTY) || diff --git a/include/linux/f2fs_fs.h b/include/linux/f2fs_fs.h index 105cfeedea74..53344e1f2b44 100644 --- a/include/linux/f2fs_fs.h +++ b/include/linux/f2fs_fs.h @@ -34,7 +34,6 @@ #define F2FS_RESERVED_NODE_NUM 3 #define F2FS_ROOT_INO(sbi) ((sbi)->root_ino_num) -#define F2FS_COMPRESS_INO(sbi) (NM_I(sbi)->max_nid) #define F2FS_MAX_QUOTAS 3 -- 2.49.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [PATCH v1 10/12] f2fs: cache: support fault injection 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel -1 siblings, 0 replies; 34+ messages in thread From: Chao Yu @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-f2fs-devel, linux-kernel, Chao Yu This patch adds fault injection support for metadata cache allocations to improve error-path test coverage. - it integrates entry allocations with FAULT_KALLOC Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index 173bbf3aa94c..c1e5c945391f 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -175,20 +175,24 @@ static struct f2fs_cached_block *f2fs_create_cache( struct f2fs_cached_block_list *cache, unsigned long index, bool nofail) { + struct f2fs_sb_info *sbi = cache->sbi; struct f2fs_cached_block *entry; unsigned int flags = GFP_NOFS; - if (nofail) + if (nofail) { flags |= __GFP_NOFAIL; - - entry = kzalloc_obj(*entry, flags); - if (!entry) - return ERR_PTR(-ENOMEM); - - entry->data = kzalloc(cache->sbi->blocksize, flags); - if (!entry->data) { - kfree(entry); - return ERR_PTR(-ENOMEM); + entry = kzalloc_obj(*entry, flags); + entry->data = kzalloc(sbi->blocksize, flags); + } else { + entry = f2fs_kzalloc(sbi, sizeof(*entry), flags); + if (!entry) + return ERR_PTR(-ENOMEM); + + entry->data = f2fs_kzalloc(sbi, sbi->blocksize, flags); + if (!entry->data) { + kfree(entry); + return ERR_PTR(-ENOMEM); + } } entry->index = index; -- 2.49.0 ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [f2fs-dev] [PATCH v1 10/12] f2fs: cache: support fault injection @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-kernel, linux-f2fs-devel This patch adds fault injection support for metadata cache allocations to improve error-path test coverage. - it integrates entry allocations with FAULT_KALLOC Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index 173bbf3aa94c..c1e5c945391f 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -175,20 +175,24 @@ static struct f2fs_cached_block *f2fs_create_cache( struct f2fs_cached_block_list *cache, unsigned long index, bool nofail) { + struct f2fs_sb_info *sbi = cache->sbi; struct f2fs_cached_block *entry; unsigned int flags = GFP_NOFS; - if (nofail) + if (nofail) { flags |= __GFP_NOFAIL; - - entry = kzalloc_obj(*entry, flags); - if (!entry) - return ERR_PTR(-ENOMEM); - - entry->data = kzalloc(cache->sbi->blocksize, flags); - if (!entry->data) { - kfree(entry); - return ERR_PTR(-ENOMEM); + entry = kzalloc_obj(*entry, flags); + entry->data = kzalloc(sbi->blocksize, flags); + } else { + entry = f2fs_kzalloc(sbi, sizeof(*entry), flags); + if (!entry) + return ERR_PTR(-ENOMEM); + + entry->data = f2fs_kzalloc(sbi, sbi->blocksize, flags); + if (!entry->data) { + kfree(entry); + return ERR_PTR(-ENOMEM); + } } entry->index = index; -- 2.49.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [PATCH v1 11/12] f2fs: cache: introduce tracepoints 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel -1 siblings, 0 replies; 34+ messages in thread From: Chao Yu @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-f2fs-devel, linux-kernel, Chao Yu This patch introduces ftrace tracepoints to observe and profile metadata cache operations: - trace_f2fs_cache_set_dirty to trace marking a cached block dirty - trace_f2fs_write_cache to trace single block writeback submission - trace_f2fs_write_caches to tracesbatch writeback and sync sessions Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 2 ++ fs/f2fs/checkpoint.c | 6 ++++ fs/f2fs/segment.c | 3 ++ include/trace/events/f2fs.h | 71 +++++++++++++++++++++++++++++++++++++ 4 files changed, 82 insertions(+) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index c1e5c945391f..d57adf52c3a5 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -16,6 +16,7 @@ #include "f2fs.h" #include "cache.h" #include "node.h" +#include <trace/events/f2fs.h> #include "segment.h" void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, @@ -78,6 +79,7 @@ bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry) enum count_type type = IS_META_CACHE(cache) ? F2FS_DIRTY_META : F2FS_DIRTY_NODES; + trace_f2fs_cache_set_dirty(entry, IS_META_CACHE(cache) ? META : NODE); f2fs_cache_update_tag(entry, 0, F2FS_CACHE_TAG_DIRTY); inc_page_count(cache->sbi, type); return true; diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c index 508132652693..8bb3b701ce89 100644 --- a/fs/f2fs/checkpoint.c +++ b/fs/f2fs/checkpoint.c @@ -550,6 +550,8 @@ static bool __f2fs_write_meta_cache(struct f2fs_cached_block *entry, { struct f2fs_sb_info *sbi = entry->cache->sbi; + trace_f2fs_write_cache(entry, META); + if (unlikely(f2fs_cp_error(sbi))) { if (is_sbi_flag_set(sbi, SBI_IS_CLOSE)) { f2fs_force_clear_cache_dirty(entry); @@ -608,6 +610,8 @@ long f2fs_sync_meta_caches(struct f2fs_sb_info *sbi, long nr_to_write, struct blk_plug plug; bool background = nr_to_write != LONG_MAX; + trace_f2fs_write_caches(sbi, nr_to_write, 0, META); + blk_start_plug(&plug); while ((nr = f2fs_cache_gang_lookup_tag(META_CACHE(sbi), entries, @@ -663,6 +667,8 @@ long f2fs_sync_meta_caches(struct f2fs_sb_info *sbi, long nr_to_write, blk_finish_plug(&plug); + trace_f2fs_write_caches(sbi, nr_to_write, nwritten, META); + return nwritten; } diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c index 3a585fc7c197..e37d57052b49 100644 --- a/fs/f2fs/segment.c +++ b/fs/f2fs/segment.c @@ -4170,6 +4170,9 @@ void f2fs_do_write_node_page(unsigned int nid, struct f2fs_io_info *fio) { struct f2fs_summary sum; + if (fio->is_cache) + trace_f2fs_write_cache(fio->cache_entry, NODE); + set_summary(&sum, nid, 0, 0); do_write_page(&sum, fio); diff --git a/include/trace/events/f2fs.h b/include/trace/events/f2fs.h index 1dd9fc5afc46..7eb3528d35be 100644 --- a/include/trace/events/f2fs.h +++ b/include/trace/events/f2fs.h @@ -1429,6 +1429,50 @@ DEFINE_EVENT(f2fs__folio, f2fs_set_page_dirty, TP_ARGS(folio, type) ); +DECLARE_EVENT_CLASS(f2fs__cached_block, + + TP_PROTO(struct f2fs_cached_block *block, int type), + + TP_ARGS(block, type), + + TP_STRUCT__entry( + __field(dev_t, dev) + __field(pgoff_t, index) + __field(int, type) + __field(int, dirty) + __field(int, uptodate) + ), + + TP_fast_assign( + __entry->dev = block->cache->sbi->sb->s_dev; + __entry->index = block->index; + __entry->type = type; + __entry->dirty = f2fs_cache_test_dirty(block); + __entry->uptodate = f2fs_cache_test_uptodate(block); + ), + + TP_printk("dev = (%d,%d), %s, index = %lu, dirty = %d, uptodate = %d", + show_dev(__entry->dev), + show_block_type(__entry->type), + (unsigned long)__entry->index, + __entry->dirty, + __entry->uptodate) +); + +DEFINE_EVENT(f2fs__cached_block, f2fs_write_cache, + + TP_PROTO(struct f2fs_cached_block *block, int type), + + TP_ARGS(block, type) +); + +DEFINE_EVENT(f2fs__cached_block, f2fs_cache_set_dirty, + + TP_PROTO(struct f2fs_cached_block *block, int type), + + TP_ARGS(block, type) +); + TRACE_EVENT(f2fs_replace_atomic_write_block, TP_PROTO(struct inode *inode, struct inode *cow_inode, pgoff_t index, @@ -1573,6 +1617,33 @@ TRACE_EVENT(f2fs_writepages, __entry->for_sync) ); +TRACE_EVENT(f2fs_write_caches, + + TP_PROTO(struct f2fs_sb_info *sbi, long nr_to_write, long nwritten, int type), + + TP_ARGS(sbi, nr_to_write, nwritten, type), + + TP_STRUCT__entry( + __field(dev_t, dev) + __field(long, nr_to_write) + __field(long, nwritten) + __field(int, type) + ), + + TP_fast_assign( + __entry->dev = sbi->sb->s_dev; + __entry->nr_to_write = nr_to_write; + __entry->nwritten = nwritten; + __entry->type = type; + ), + + TP_printk("dev = (%d,%d), %s, nr_to_write = %ld, nwritten = %ld", + show_dev(__entry->dev), + show_block_type(__entry->type), + __entry->nr_to_write, + __entry->nwritten) +); + TRACE_EVENT(f2fs_readpages, TP_PROTO(struct inode *inode, pgoff_t start, unsigned int nrpage), -- 2.49.0 ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [f2fs-dev] [PATCH v1 11/12] f2fs: cache: introduce tracepoints @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-kernel, linux-f2fs-devel This patch introduces ftrace tracepoints to observe and profile metadata cache operations: - trace_f2fs_cache_set_dirty to trace marking a cached block dirty - trace_f2fs_write_cache to trace single block writeback submission - trace_f2fs_write_caches to tracesbatch writeback and sync sessions Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.c | 2 ++ fs/f2fs/checkpoint.c | 6 ++++ fs/f2fs/segment.c | 3 ++ include/trace/events/f2fs.h | 71 +++++++++++++++++++++++++++++++++++++ 4 files changed, 82 insertions(+) diff --git a/fs/f2fs/cache.c b/fs/f2fs/cache.c index c1e5c945391f..d57adf52c3a5 100644 --- a/fs/f2fs/cache.c +++ b/fs/f2fs/cache.c @@ -16,6 +16,7 @@ #include "f2fs.h" #include "cache.h" #include "node.h" +#include <trace/events/f2fs.h> #include "segment.h" void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry, @@ -78,6 +79,7 @@ bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry) enum count_type type = IS_META_CACHE(cache) ? F2FS_DIRTY_META : F2FS_DIRTY_NODES; + trace_f2fs_cache_set_dirty(entry, IS_META_CACHE(cache) ? META : NODE); f2fs_cache_update_tag(entry, 0, F2FS_CACHE_TAG_DIRTY); inc_page_count(cache->sbi, type); return true; diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c index 508132652693..8bb3b701ce89 100644 --- a/fs/f2fs/checkpoint.c +++ b/fs/f2fs/checkpoint.c @@ -550,6 +550,8 @@ static bool __f2fs_write_meta_cache(struct f2fs_cached_block *entry, { struct f2fs_sb_info *sbi = entry->cache->sbi; + trace_f2fs_write_cache(entry, META); + if (unlikely(f2fs_cp_error(sbi))) { if (is_sbi_flag_set(sbi, SBI_IS_CLOSE)) { f2fs_force_clear_cache_dirty(entry); @@ -608,6 +610,8 @@ long f2fs_sync_meta_caches(struct f2fs_sb_info *sbi, long nr_to_write, struct blk_plug plug; bool background = nr_to_write != LONG_MAX; + trace_f2fs_write_caches(sbi, nr_to_write, 0, META); + blk_start_plug(&plug); while ((nr = f2fs_cache_gang_lookup_tag(META_CACHE(sbi), entries, @@ -663,6 +667,8 @@ long f2fs_sync_meta_caches(struct f2fs_sb_info *sbi, long nr_to_write, blk_finish_plug(&plug); + trace_f2fs_write_caches(sbi, nr_to_write, nwritten, META); + return nwritten; } diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c index 3a585fc7c197..e37d57052b49 100644 --- a/fs/f2fs/segment.c +++ b/fs/f2fs/segment.c @@ -4170,6 +4170,9 @@ void f2fs_do_write_node_page(unsigned int nid, struct f2fs_io_info *fio) { struct f2fs_summary sum; + if (fio->is_cache) + trace_f2fs_write_cache(fio->cache_entry, NODE); + set_summary(&sum, nid, 0, 0); do_write_page(&sum, fio); diff --git a/include/trace/events/f2fs.h b/include/trace/events/f2fs.h index 1dd9fc5afc46..7eb3528d35be 100644 --- a/include/trace/events/f2fs.h +++ b/include/trace/events/f2fs.h @@ -1429,6 +1429,50 @@ DEFINE_EVENT(f2fs__folio, f2fs_set_page_dirty, TP_ARGS(folio, type) ); +DECLARE_EVENT_CLASS(f2fs__cached_block, + + TP_PROTO(struct f2fs_cached_block *block, int type), + + TP_ARGS(block, type), + + TP_STRUCT__entry( + __field(dev_t, dev) + __field(pgoff_t, index) + __field(int, type) + __field(int, dirty) + __field(int, uptodate) + ), + + TP_fast_assign( + __entry->dev = block->cache->sbi->sb->s_dev; + __entry->index = block->index; + __entry->type = type; + __entry->dirty = f2fs_cache_test_dirty(block); + __entry->uptodate = f2fs_cache_test_uptodate(block); + ), + + TP_printk("dev = (%d,%d), %s, index = %lu, dirty = %d, uptodate = %d", + show_dev(__entry->dev), + show_block_type(__entry->type), + (unsigned long)__entry->index, + __entry->dirty, + __entry->uptodate) +); + +DEFINE_EVENT(f2fs__cached_block, f2fs_write_cache, + + TP_PROTO(struct f2fs_cached_block *block, int type), + + TP_ARGS(block, type) +); + +DEFINE_EVENT(f2fs__cached_block, f2fs_cache_set_dirty, + + TP_PROTO(struct f2fs_cached_block *block, int type), + + TP_ARGS(block, type) +); + TRACE_EVENT(f2fs_replace_atomic_write_block, TP_PROTO(struct inode *inode, struct inode *cow_inode, pgoff_t index, @@ -1573,6 +1617,33 @@ TRACE_EVENT(f2fs_writepages, __entry->for_sync) ); +TRACE_EVENT(f2fs_write_caches, + + TP_PROTO(struct f2fs_sb_info *sbi, long nr_to_write, long nwritten, int type), + + TP_ARGS(sbi, nr_to_write, nwritten, type), + + TP_STRUCT__entry( + __field(dev_t, dev) + __field(long, nr_to_write) + __field(long, nwritten) + __field(int, type) + ), + + TP_fast_assign( + __entry->dev = sbi->sb->s_dev; + __entry->nr_to_write = nr_to_write; + __entry->nwritten = nwritten; + __entry->type = type; + ), + + TP_printk("dev = (%d,%d), %s, nr_to_write = %ld, nwritten = %ld", + show_dev(__entry->dev), + show_block_type(__entry->type), + __entry->nr_to_write, + __entry->nwritten) +); + TRACE_EVENT(f2fs_readpages, TP_PROTO(struct inode *inode, pgoff_t start, unsigned int nrpage), -- 2.49.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [PATCH v1 12/12] f2fs: cache: show per-cache usage in debugfs 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel -1 siblings, 0 replies; 34+ messages in thread From: Chao Yu @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-f2fs-devel, linux-kernel, Chao Yu This patch exposes per-cache memory usage (entry struct memory vs. cached block buffer memory) for meta, node, and compress caches under the memory section in debugfs: - meta entry: xxx KB, meta cache: xxx KB - node entry: xxx KB, node cache: xxx KB - compress entry: xxx KB, compress cache: xxx KB Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.h | 1 + fs/f2fs/debug.c | 49 ++++++++++++++++++++++++++++++++++++++++--------- fs/f2fs/f2fs.h | 2 ++ 3 files changed, 43 insertions(+), 9 deletions(-) diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index 12c061d28d23..dccfc90fbab1 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -39,6 +39,7 @@ enum f2fs_cache_type { F2FS_META_CACHE, F2FS_NODE_CACHE, F2FS_COMPRESS_CACHE, + NR_CACHE_TYPES, }; /* Main cache control structure (per sb_info) */ diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c index 8cd06f7ba9e7..68a9d589da8c 100644 --- a/fs/f2fs/debug.c +++ b/fs/f2fs/debug.c @@ -380,17 +380,36 @@ static void update_mem_info(struct f2fs_sb_info *sbi) si->cache_mem += si->ext_mem[i]; } - si->page_mem = 0; - si->page_mem += (unsigned long long)META_CACHE(sbi)->num_entries << PAGE_SHIFT; - si->cache_mem += META_CACHE(sbi)->num_entries * sizeof(struct f2fs_cached_block); - si->page_mem += (unsigned long long)NODE_CACHE(sbi)->num_entries << PAGE_SHIFT; - si->cache_mem += NODE_CACHE(sbi)->num_entries * sizeof(struct f2fs_cached_block); + si->cache_entry_mem[F2FS_META_CACHE] = + (unsigned long long)META_CACHE(sbi)->num_entries * + sizeof(struct f2fs_cached_block); + si->cache_data_mem[F2FS_META_CACHE] = + (unsigned long long)META_CACHE(sbi)->num_entries * sbi->blocksize; + + si->cache_entry_mem[F2FS_NODE_CACHE] = + (unsigned long long)NODE_CACHE(sbi)->num_entries * + sizeof(struct f2fs_cached_block); + si->cache_data_mem[F2FS_NODE_CACHE] = + (unsigned long long)NODE_CACHE(sbi)->num_entries * sbi->blocksize; + + si->cache_mem += si->cache_entry_mem[F2FS_META_CACHE] + + si->cache_entry_mem[F2FS_NODE_CACHE]; + si->page_mem = si->cache_data_mem[F2FS_META_CACHE] + + si->cache_data_mem[F2FS_NODE_CACHE]; #ifdef CONFIG_F2FS_FS_COMPRESSION if (test_opt(sbi, COMPRESS_CACHE)) { - unsigned long npages = COMPRESS_CACHE(sbi)->num_entries; - - si->page_mem += (unsigned long long)npages << PAGE_SHIFT; - si->cache_mem += npages * sizeof(struct f2fs_cached_block); + si->cache_entry_mem[F2FS_COMPRESS_CACHE] = + (unsigned long long)COMPRESS_CACHE(sbi)->num_entries * + sizeof(struct f2fs_cached_block); + si->cache_data_mem[F2FS_COMPRESS_CACHE] = + (unsigned long long)COMPRESS_CACHE(sbi)->num_entries * + sbi->blocksize; + + si->cache_mem += si->cache_entry_mem[F2FS_COMPRESS_CACHE]; + si->page_mem += si->cache_data_mem[F2FS_COMPRESS_CACHE]; + } else { + si->cache_entry_mem[F2FS_COMPRESS_CACHE] = 0; + si->cache_data_mem[F2FS_COMPRESS_CACHE] = 0; } #endif } @@ -749,6 +768,18 @@ static int stat_show(struct seq_file *s, void *v) si->ext_mem[EX_READ] >> 10); seq_printf(s, " - block age extent cache: %llu KB\n", si->ext_mem[EX_BLOCK_AGE] >> 10); + seq_printf(s, " - meta entry: %llu KB, meta cache: %llu KB\n", + si->cache_entry_mem[F2FS_META_CACHE] >> 10, + si->cache_data_mem[F2FS_META_CACHE] >> 10); + seq_printf(s, " - node entry: %llu KB, node cache: %llu KB\n", + si->cache_entry_mem[F2FS_NODE_CACHE] >> 10, + si->cache_data_mem[F2FS_NODE_CACHE] >> 10); +#ifdef CONFIG_F2FS_FS_COMPRESSION + if (test_opt(sbi, COMPRESS_CACHE)) + seq_printf(s, " - compress entry: %llu KB, compress cache: %llu KB\n", + si->cache_entry_mem[F2FS_COMPRESS_CACHE] >> 10, + si->cache_data_mem[F2FS_COMPRESS_CACHE] >> 10); +#endif seq_printf(s, " - paged : %llu KB\n", si->page_mem >> 10); } diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 0a749a59f386..42952dde79e0 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -4453,6 +4453,8 @@ struct f2fs_stat_info { unsigned int block_count[2]; unsigned int inplace_count; unsigned long long base_mem, cache_mem, page_mem; + unsigned long long cache_entry_mem[NR_CACHE_TYPES]; + unsigned long long cache_data_mem[NR_CACHE_TYPES]; struct f2fs_dev_stats *dev_stats; }; -- 2.49.0 ^ permalink raw reply related [flat|nested] 34+ messages in thread
* [f2fs-dev] [PATCH v1 12/12] f2fs: cache: show per-cache usage in debugfs @ 2026-08-20 3:17 ` Chao Yu via Linux-f2fs-devel 0 siblings, 0 replies; 34+ messages in thread From: Chao Yu via Linux-f2fs-devel @ 2026-08-20 3:17 UTC (permalink / raw) To: jaegeuk; +Cc: linux-kernel, linux-f2fs-devel This patch exposes per-cache memory usage (entry struct memory vs. cached block buffer memory) for meta, node, and compress caches under the memory section in debugfs: - meta entry: xxx KB, meta cache: xxx KB - node entry: xxx KB, node cache: xxx KB - compress entry: xxx KB, compress cache: xxx KB Signed-off-by: Chao Yu <chao@kernel.org> --- fs/f2fs/cache.h | 1 + fs/f2fs/debug.c | 49 ++++++++++++++++++++++++++++++++++++++++--------- fs/f2fs/f2fs.h | 2 ++ 3 files changed, 43 insertions(+), 9 deletions(-) diff --git a/fs/f2fs/cache.h b/fs/f2fs/cache.h index 12c061d28d23..dccfc90fbab1 100644 --- a/fs/f2fs/cache.h +++ b/fs/f2fs/cache.h @@ -39,6 +39,7 @@ enum f2fs_cache_type { F2FS_META_CACHE, F2FS_NODE_CACHE, F2FS_COMPRESS_CACHE, + NR_CACHE_TYPES, }; /* Main cache control structure (per sb_info) */ diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c index 8cd06f7ba9e7..68a9d589da8c 100644 --- a/fs/f2fs/debug.c +++ b/fs/f2fs/debug.c @@ -380,17 +380,36 @@ static void update_mem_info(struct f2fs_sb_info *sbi) si->cache_mem += si->ext_mem[i]; } - si->page_mem = 0; - si->page_mem += (unsigned long long)META_CACHE(sbi)->num_entries << PAGE_SHIFT; - si->cache_mem += META_CACHE(sbi)->num_entries * sizeof(struct f2fs_cached_block); - si->page_mem += (unsigned long long)NODE_CACHE(sbi)->num_entries << PAGE_SHIFT; - si->cache_mem += NODE_CACHE(sbi)->num_entries * sizeof(struct f2fs_cached_block); + si->cache_entry_mem[F2FS_META_CACHE] = + (unsigned long long)META_CACHE(sbi)->num_entries * + sizeof(struct f2fs_cached_block); + si->cache_data_mem[F2FS_META_CACHE] = + (unsigned long long)META_CACHE(sbi)->num_entries * sbi->blocksize; + + si->cache_entry_mem[F2FS_NODE_CACHE] = + (unsigned long long)NODE_CACHE(sbi)->num_entries * + sizeof(struct f2fs_cached_block); + si->cache_data_mem[F2FS_NODE_CACHE] = + (unsigned long long)NODE_CACHE(sbi)->num_entries * sbi->blocksize; + + si->cache_mem += si->cache_entry_mem[F2FS_META_CACHE] + + si->cache_entry_mem[F2FS_NODE_CACHE]; + si->page_mem = si->cache_data_mem[F2FS_META_CACHE] + + si->cache_data_mem[F2FS_NODE_CACHE]; #ifdef CONFIG_F2FS_FS_COMPRESSION if (test_opt(sbi, COMPRESS_CACHE)) { - unsigned long npages = COMPRESS_CACHE(sbi)->num_entries; - - si->page_mem += (unsigned long long)npages << PAGE_SHIFT; - si->cache_mem += npages * sizeof(struct f2fs_cached_block); + si->cache_entry_mem[F2FS_COMPRESS_CACHE] = + (unsigned long long)COMPRESS_CACHE(sbi)->num_entries * + sizeof(struct f2fs_cached_block); + si->cache_data_mem[F2FS_COMPRESS_CACHE] = + (unsigned long long)COMPRESS_CACHE(sbi)->num_entries * + sbi->blocksize; + + si->cache_mem += si->cache_entry_mem[F2FS_COMPRESS_CACHE]; + si->page_mem += si->cache_data_mem[F2FS_COMPRESS_CACHE]; + } else { + si->cache_entry_mem[F2FS_COMPRESS_CACHE] = 0; + si->cache_data_mem[F2FS_COMPRESS_CACHE] = 0; } #endif } @@ -749,6 +768,18 @@ static int stat_show(struct seq_file *s, void *v) si->ext_mem[EX_READ] >> 10); seq_printf(s, " - block age extent cache: %llu KB\n", si->ext_mem[EX_BLOCK_AGE] >> 10); + seq_printf(s, " - meta entry: %llu KB, meta cache: %llu KB\n", + si->cache_entry_mem[F2FS_META_CACHE] >> 10, + si->cache_data_mem[F2FS_META_CACHE] >> 10); + seq_printf(s, " - node entry: %llu KB, node cache: %llu KB\n", + si->cache_entry_mem[F2FS_NODE_CACHE] >> 10, + si->cache_data_mem[F2FS_NODE_CACHE] >> 10); +#ifdef CONFIG_F2FS_FS_COMPRESSION + if (test_opt(sbi, COMPRESS_CACHE)) + seq_printf(s, " - compress entry: %llu KB, compress cache: %llu KB\n", + si->cache_entry_mem[F2FS_COMPRESS_CACHE] >> 10, + si->cache_data_mem[F2FS_COMPRESS_CACHE] >> 10); +#endif seq_printf(s, " - paged : %llu KB\n", si->page_mem >> 10); } diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 0a749a59f386..42952dde79e0 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -4453,6 +4453,8 @@ struct f2fs_stat_info { unsigned int block_count[2]; unsigned int inplace_count; unsigned long long base_mem, cache_mem, page_mem; + unsigned long long cache_entry_mem[NR_CACHE_TYPES]; + unsigned long long cache_data_mem[NR_CACHE_TYPES]; struct f2fs_dev_stats *dev_stats; }; -- 2.49.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 34+ messages in thread
end of thread, other threads:[~2026-08-20 5:13 UTC | newest] Thread overview: 34+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-08-20 3:17 [PATCH v1 00/12] f2fs: introduce metadata cache Chao Yu 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel 2026-08-20 3:17 ` [PATCH v1 01/12] f2fs: cache: implement " Chao Yu 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel 2026-08-20 5:08 ` Jaegeuk Kim 2026-08-20 5:08 ` Jaegeuk Kim via Linux-f2fs-devel 2026-08-20 5:13 ` Chao Yu 2026-08-20 5:13 ` Chao Yu via Linux-f2fs-devel 2026-08-20 3:17 ` [PATCH v1 02/12] f2fs: cache: initialize meta cache Chao Yu 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel 2026-08-20 3:17 ` [PATCH v1 03/12] f2fs: cache: introduce shrinker Chao Yu 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel 2026-08-20 3:17 ` [PATCH v1 04/12] f2fs: cache: introduce writeback thread Chao Yu 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel 2026-08-20 5:09 ` Jaegeuk Kim 2026-08-20 5:09 ` Jaegeuk Kim via Linux-f2fs-devel 2026-08-20 3:17 ` [f2fs-dev] [PATCH v1 05/12] f2fs: cache: use meta cache Chao Yu via Linux-f2fs-devel 2026-08-20 3:17 ` Chao Yu 2026-08-20 5:11 ` [f2fs-dev] " Jaegeuk Kim via Linux-f2fs-devel 2026-08-20 5:11 ` Jaegeuk Kim 2026-08-20 3:17 ` [PATCH v1 06/12] f2fs: cache: initialize node cache Chao Yu 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel 2026-08-20 3:17 ` [PATCH v1 07/12] f2fs: cache: use " Chao Yu 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel 2026-08-20 3:17 ` [PATCH v1 08/12] f2fs: cache: initialize compress cache Chao Yu 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel 2026-08-20 3:17 ` [PATCH v1 09/12] f2fs: cache: use " Chao Yu 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel 2026-08-20 3:17 ` [PATCH v1 10/12] f2fs: cache: support fault injection Chao Yu 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel 2026-08-20 3:17 ` [PATCH v1 11/12] f2fs: cache: introduce tracepoints Chao Yu 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel 2026-08-20 3:17 ` [PATCH v1 12/12] f2fs: cache: show per-cache usage in debugfs Chao Yu 2026-08-20 3:17 ` [f2fs-dev] " Chao Yu via Linux-f2fs-devel
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.