* [PATCH RFC PKS/PMEM 24/58] fs/freevxfs: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
cluster-devel, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, linux-afs, Christoph Hellwig, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-cifs,
linux-um, intel-gfx, ecryptfs, linux-erofs, reiserfs-devel,
linux-block, linux-bcache, Dan Williams, io-uring, linux-nfs,
linux-ntfs-dev, netdev, kexec, linux-kernel, linux-f2fs-devel,
linux-fsdevel, bpf, linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this FS are localized to a single thread. To avoid
the over head of global PKRS updates use the new kmap_thread() call.
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
fs/freevxfs/vxfs_immed.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/fs/freevxfs/vxfs_immed.c b/fs/freevxfs/vxfs_immed.c
index bfc780c682fb..9c42fec4cd85 100644
--- a/fs/freevxfs/vxfs_immed.c
+++ b/fs/freevxfs/vxfs_immed.c
@@ -69,9 +69,9 @@ vxfs_immed_readpage(struct file *fp, struct page *pp)
u_int64_t offset = (u_int64_t)pp->index << PAGE_SHIFT;
caddr_t kaddr;
- kaddr = kmap(pp);
+ kaddr = kmap_thread(pp);
memcpy(kaddr, vip->vii_immed.vi_immed + offset, PAGE_SIZE);
- kunmap(pp);
+ kunmap_thread(pp);
flush_dcache_page(pp);
SetPageUptodate(pp);
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 23/58] fs/fuse: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, Miklos Szeredi, linux-rdma,
x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-um,
intel-gfx, ecryptfs, linux-erofs, reiserfs-devel, linux-block,
linux-bcache, Dan Williams, io-uring, linux-nfs, linux-ntfs-dev,
netdev, linux-nvdimm, kexec, linux-kernel, linux-f2fs-devel,
linux-fsdevel, bpf, linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this FS are localized to a single thread. To avoid
the over head of global PKRS updates use the new kmap_thread() call.
Cc: Miklos Szeredi <miklos@szeredi.hu>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
fs/fuse/readdir.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/fs/fuse/readdir.c b/fs/fuse/readdir.c
index 90e3f01bd796..953ffe6f56e3 100644
--- a/fs/fuse/readdir.c
+++ b/fs/fuse/readdir.c
@@ -536,9 +536,9 @@ static int fuse_readdir_cached(struct file *file, struct dir_context *ctx)
* Contents of the page are now protected against changing by holding
* the page lock.
*/
- addr = kmap(page);
+ addr = kmap_thread(page);
res = fuse_parse_cache(ff, addr, size, ctx);
- kunmap(page);
+ kunmap_thread(page);
unlock_page(page);
put_page(page);
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 22/58] fs/f2fs: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, ecryptfs,
linux-um, intel-gfx, Chao Yu, linux-erofs, reiserfs-devel,
linux-block, linux-bcache, Jaegeuk Kim, Dan Williams, io-uring,
linux-nfs, linux-ntfs-dev, netdev, kexec, linux-kernel,
linux-f2fs-devel, linux-fsdevel, bpf, linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this FS are localized to a single thread. To avoid
the over head of global PKRS updates use the new kmap_thread() call.
Cc: Jaegeuk Kim <jaegeuk@kernel.org>
Cc: Chao Yu <chao@kernel.org>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
fs/f2fs/f2fs.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index d9e52a7f3702..ff72a45a577e 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -2410,12 +2410,12 @@ static inline struct page *f2fs_pagecache_get_page(
static inline void f2fs_copy_page(struct page *src, struct page *dst)
{
- char *src_kaddr = kmap(src);
- char *dst_kaddr = kmap(dst);
+ char *src_kaddr = kmap_thread(src);
+ char *dst_kaddr = kmap_thread(dst);
memcpy(dst_kaddr, src_kaddr, PAGE_SIZE);
- kunmap(dst);
- kunmap(src);
+ kunmap_thread(dst);
+ kunmap_thread(src);
}
static inline void f2fs_put_page(struct page *page, int unlock)
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 21/58] fs/nfs: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, io-uring, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-afs,
linux-um, intel-gfx, ecryptfs, linux-erofs, reiserfs-devel,
linux-block, linux-bcache, Dan Williams, Trond Myklebust,
linux-nfs, linux-ntfs-dev, netdev, kexec, linux-kernel,
linux-f2fs-devel, linux-fsdevel, bpf, linuxppc-dev,
Anna Schumaker, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this FS are localized to a single thread. To avoid
the over head of global PKRS updates use the new kmap_thread() call.
Cc: Trond Myklebust <trond.myklebust@hammerspace.com>
Cc: Anna Schumaker <anna.schumaker@netapp.com>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
fs/nfs/dir.c | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c
index cb52db9a0cfb..fee321acccb4 100644
--- a/fs/nfs/dir.c
+++ b/fs/nfs/dir.c
@@ -213,7 +213,7 @@ int nfs_readdir_make_qstr(struct qstr *string, const char *name, unsigned int le
static
int nfs_readdir_add_to_array(struct nfs_entry *entry, struct page *page)
{
- struct nfs_cache_array *array = kmap(page);
+ struct nfs_cache_array *array = kmap_thread(page);
struct nfs_cache_array_entry *cache_entry;
int ret;
@@ -235,7 +235,7 @@ int nfs_readdir_add_to_array(struct nfs_entry *entry, struct page *page)
if (entry->eof != 0)
array->eof_index = array->size;
out:
- kunmap(page);
+ kunmap_thread(page);
return ret;
}
@@ -347,7 +347,7 @@ int nfs_readdir_search_array(nfs_readdir_descriptor_t *desc)
struct nfs_cache_array *array;
int status;
- array = kmap(desc->page);
+ array = kmap_thread(desc->page);
if (*desc->dir_cookie == 0)
status = nfs_readdir_search_for_pos(array, desc);
@@ -359,7 +359,7 @@ int nfs_readdir_search_array(nfs_readdir_descriptor_t *desc)
desc->current_index += array->size;
desc->page_index++;
}
- kunmap(desc->page);
+ kunmap_thread(desc->page);
return status;
}
@@ -602,10 +602,10 @@ int nfs_readdir_page_filler(nfs_readdir_descriptor_t *desc, struct nfs_entry *en
out_nopages:
if (count == 0 || (status == -EBADCOOKIE && entry->eof != 0)) {
- array = kmap(page);
+ array = kmap_thread(page);
array->eof_index = array->size;
status = 0;
- kunmap(page);
+ kunmap_thread(page);
}
put_page(scratch);
@@ -669,7 +669,7 @@ int nfs_readdir_xdr_to_array(nfs_readdir_descriptor_t *desc, struct page *page,
goto out;
}
- array = kmap(page);
+ array = kmap_thread(page);
status = nfs_readdir_alloc_pages(pages, array_size);
if (status < 0)
@@ -691,7 +691,7 @@ int nfs_readdir_xdr_to_array(nfs_readdir_descriptor_t *desc, struct page *page,
nfs_readdir_free_pages(pages, array_size);
out_release_array:
- kunmap(page);
+ kunmap_thread(page);
nfs4_label_free(entry.label);
out:
nfs_free_fattr(entry.fattr);
@@ -803,7 +803,7 @@ int nfs_do_filldir(nfs_readdir_descriptor_t *desc)
struct nfs_cache_array *array = NULL;
struct nfs_open_dir_context *ctx = file->private_data;
- array = kmap(desc->page);
+ array = kmap_thread(desc->page);
for (i = desc->cache_entry_index; i < array->size; i++) {
struct nfs_cache_array_entry *ent;
@@ -827,7 +827,7 @@ int nfs_do_filldir(nfs_readdir_descriptor_t *desc)
if (array->eof_index >= 0)
desc->eof = true;
- kunmap(desc->page);
+ kunmap_thread(desc->page);
dfprintk(DIRCACHE, "NFS: nfs_do_filldir() filling ended @ cookie %Lu; returning = %d\n",
(unsigned long long)*desc->dir_cookie, res);
return res;
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 20/58] fs/jffs2: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
Richard Weinberger, x86, amd-gfx, linux-afs, cluster-devel,
linux-cachefs, intel-wired-lan, xen-devel, linux-ext4,
David Woodhouse, Fenghua Yu, linux-um, intel-gfx, ecryptfs,
linux-erofs, reiserfs-devel, linux-block, linux-bcache,
Dan Williams, io-uring, linux-nfs, linux-ntfs-dev, netdev, kexec,
linux-kernel, linux-f2fs-devel, linux-fsdevel, bpf, linuxppc-dev,
linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this FS are localized to a single thread. To avoid
the over head of global PKRS updates use the new kmap_thread() call.
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Richard Weinberger <richard@nod.at>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
fs/jffs2/file.c | 4 ++--
fs/jffs2/gc.c | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/fs/jffs2/file.c b/fs/jffs2/file.c
index f8fb89b10227..3e6d54f9b011 100644
--- a/fs/jffs2/file.c
+++ b/fs/jffs2/file.c
@@ -88,7 +88,7 @@ static int jffs2_do_readpage_nolock (struct inode *inode, struct page *pg)
BUG_ON(!PageLocked(pg));
- pg_buf = kmap(pg);
+ pg_buf = kmap_thread(pg);
/* FIXME: Can kmap fail? */
ret = jffs2_read_inode_range(c, f, pg_buf, pg->index << PAGE_SHIFT,
@@ -103,7 +103,7 @@ static int jffs2_do_readpage_nolock (struct inode *inode, struct page *pg)
}
flush_dcache_page(pg);
- kunmap(pg);
+ kunmap_thread(pg);
jffs2_dbg(2, "readpage finished\n");
return ret;
diff --git a/fs/jffs2/gc.c b/fs/jffs2/gc.c
index 373b3b7c9f44..a7259783ab84 100644
--- a/fs/jffs2/gc.c
+++ b/fs/jffs2/gc.c
@@ -1335,7 +1335,7 @@ static int jffs2_garbage_collect_dnode(struct jffs2_sb_info *c, struct jffs2_era
return PTR_ERR(page);
}
- pg_ptr = kmap(page);
+ pg_ptr = kmap_thread(page);
mutex_lock(&f->sem);
offset = start;
@@ -1400,7 +1400,7 @@ static int jffs2_garbage_collect_dnode(struct jffs2_sb_info *c, struct jffs2_era
}
}
- kunmap(page);
+ kunmap_thread(page);
put_page(page);
return ret;
}
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 19/58] fs/hfsplus: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-um,
intel-gfx, ecryptfs, linux-erofs, reiserfs-devel, linux-block,
linux-bcache, Dan Williams, io-uring, linux-nfs, linux-ntfs-dev,
netdev, kexec, linux-kernel, linux-f2fs-devel, linux-fsdevel, bpf,
linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this FS are localized to a single thread. To avoid
the over head of global PKRS updates use the new kmap_thread() call.
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
fs/hfsplus/bitmap.c | 20 ++++-----
fs/hfsplus/bnode.c | 102 ++++++++++++++++++++++----------------------
fs/hfsplus/btree.c | 18 ++++----
3 files changed, 70 insertions(+), 70 deletions(-)
diff --git a/fs/hfsplus/bitmap.c b/fs/hfsplus/bitmap.c
index cebce0cfe340..9ec7c1559a0c 100644
--- a/fs/hfsplus/bitmap.c
+++ b/fs/hfsplus/bitmap.c
@@ -39,7 +39,7 @@ int hfsplus_block_allocate(struct super_block *sb, u32 size,
start = size;
goto out;
}
- pptr = kmap(page);
+ pptr = kmap_thread(page);
curr = pptr + (offset & (PAGE_CACHE_BITS - 1)) / 32;
i = offset % 32;
offset &= ~(PAGE_CACHE_BITS - 1);
@@ -74,7 +74,7 @@ int hfsplus_block_allocate(struct super_block *sb, u32 size,
}
curr++;
}
- kunmap(page);
+ kunmap_thread(page);
offset += PAGE_CACHE_BITS;
if (offset >= size)
break;
@@ -84,7 +84,7 @@ int hfsplus_block_allocate(struct super_block *sb, u32 size,
start = size;
goto out;
}
- curr = pptr = kmap(page);
+ curr = pptr = kmap_thread(page);
if ((size ^ offset) / PAGE_CACHE_BITS)
end = pptr + PAGE_CACHE_BITS / 32;
else
@@ -127,7 +127,7 @@ int hfsplus_block_allocate(struct super_block *sb, u32 size,
len -= 32;
}
set_page_dirty(page);
- kunmap(page);
+ kunmap_thread(page);
offset += PAGE_CACHE_BITS;
page = read_mapping_page(mapping, offset / PAGE_CACHE_BITS,
NULL);
@@ -135,7 +135,7 @@ int hfsplus_block_allocate(struct super_block *sb, u32 size,
start = size;
goto out;
}
- pptr = kmap(page);
+ pptr = kmap_thread(page);
curr = pptr;
end = pptr + PAGE_CACHE_BITS / 32;
}
@@ -151,7 +151,7 @@ int hfsplus_block_allocate(struct super_block *sb, u32 size,
done:
*curr = cpu_to_be32(n);
set_page_dirty(page);
- kunmap(page);
+ kunmap_thread(page);
*max = offset + (curr - pptr) * 32 + i - start;
sbi->free_blocks -= *max;
hfsplus_mark_mdb_dirty(sb);
@@ -185,7 +185,7 @@ int hfsplus_block_free(struct super_block *sb, u32 offset, u32 count)
page = read_mapping_page(mapping, pnr, NULL);
if (IS_ERR(page))
goto kaboom;
- pptr = kmap(page);
+ pptr = kmap_thread(page);
curr = pptr + (offset & (PAGE_CACHE_BITS - 1)) / 32;
end = pptr + PAGE_CACHE_BITS / 32;
len = count;
@@ -215,11 +215,11 @@ int hfsplus_block_free(struct super_block *sb, u32 offset, u32 count)
if (!count)
break;
set_page_dirty(page);
- kunmap(page);
+ kunmap_thread(page);
page = read_mapping_page(mapping, ++pnr, NULL);
if (IS_ERR(page))
goto kaboom;
- pptr = kmap(page);
+ pptr = kmap_thread(page);
curr = pptr;
end = pptr + PAGE_CACHE_BITS / 32;
}
@@ -231,7 +231,7 @@ int hfsplus_block_free(struct super_block *sb, u32 offset, u32 count)
}
out:
set_page_dirty(page);
- kunmap(page);
+ kunmap_thread(page);
sbi->free_blocks += len;
hfsplus_mark_mdb_dirty(sb);
mutex_unlock(&sbi->alloc_mutex);
diff --git a/fs/hfsplus/bnode.c b/fs/hfsplus/bnode.c
index 177fae4e6581..62757d92fbbd 100644
--- a/fs/hfsplus/bnode.c
+++ b/fs/hfsplus/bnode.c
@@ -29,14 +29,14 @@ void hfs_bnode_read(struct hfs_bnode *node, void *buf, int off, int len)
off &= ~PAGE_MASK;
l = min_t(int, len, PAGE_SIZE - off);
- memcpy(buf, kmap(*pagep) + off, l);
- kunmap(*pagep);
+ memcpy(buf, kmap_thread(*pagep) + off, l);
+ kunmap_thread(*pagep);
while ((len -= l) != 0) {
buf += l;
l = min_t(int, len, PAGE_SIZE);
- memcpy(buf, kmap(*++pagep), l);
- kunmap(*pagep);
+ memcpy(buf, kmap_thread(*++pagep), l);
+ kunmap_thread(*pagep);
}
}
@@ -82,16 +82,16 @@ void hfs_bnode_write(struct hfs_bnode *node, void *buf, int off, int len)
off &= ~PAGE_MASK;
l = min_t(int, len, PAGE_SIZE - off);
- memcpy(kmap(*pagep) + off, buf, l);
+ memcpy(kmap_thread(*pagep) + off, buf, l);
set_page_dirty(*pagep);
- kunmap(*pagep);
+ kunmap_thread(*pagep);
while ((len -= l) != 0) {
buf += l;
l = min_t(int, len, PAGE_SIZE);
- memcpy(kmap(*++pagep), buf, l);
+ memcpy(kmap_thread(*++pagep), buf, l);
set_page_dirty(*pagep);
- kunmap(*pagep);
+ kunmap_thread(*pagep);
}
}
@@ -112,15 +112,15 @@ void hfs_bnode_clear(struct hfs_bnode *node, int off, int len)
off &= ~PAGE_MASK;
l = min_t(int, len, PAGE_SIZE - off);
- memset(kmap(*pagep) + off, 0, l);
+ memset(kmap_thread(*pagep) + off, 0, l);
set_page_dirty(*pagep);
- kunmap(*pagep);
+ kunmap_thread(*pagep);
while ((len -= l) != 0) {
l = min_t(int, len, PAGE_SIZE);
- memset(kmap(*++pagep), 0, l);
+ memset(kmap_thread(*++pagep), 0, l);
set_page_dirty(*pagep);
- kunmap(*pagep);
+ kunmap_thread(*pagep);
}
}
@@ -142,24 +142,24 @@ void hfs_bnode_copy(struct hfs_bnode *dst_node, int dst,
if (src == dst) {
l = min_t(int, len, PAGE_SIZE - src);
- memcpy(kmap(*dst_page) + src, kmap(*src_page) + src, l);
- kunmap(*src_page);
+ memcpy(kmap_thread(*dst_page) + src, kmap_thread(*src_page) + src, l);
+ kunmap_thread(*src_page);
set_page_dirty(*dst_page);
- kunmap(*dst_page);
+ kunmap_thread(*dst_page);
while ((len -= l) != 0) {
l = min_t(int, len, PAGE_SIZE);
- memcpy(kmap(*++dst_page), kmap(*++src_page), l);
- kunmap(*src_page);
+ memcpy(kmap_thread(*++dst_page), kmap_thread(*++src_page), l);
+ kunmap_thread(*src_page);
set_page_dirty(*dst_page);
- kunmap(*dst_page);
+ kunmap_thread(*dst_page);
}
} else {
void *src_ptr, *dst_ptr;
do {
- src_ptr = kmap(*src_page) + src;
- dst_ptr = kmap(*dst_page) + dst;
+ src_ptr = kmap_thread(*src_page) + src;
+ dst_ptr = kmap_thread(*dst_page) + dst;
if (PAGE_SIZE - src < PAGE_SIZE - dst) {
l = PAGE_SIZE - src;
src = 0;
@@ -171,9 +171,9 @@ void hfs_bnode_copy(struct hfs_bnode *dst_node, int dst,
}
l = min(len, l);
memcpy(dst_ptr, src_ptr, l);
- kunmap(*src_page);
+ kunmap_thread(*src_page);
set_page_dirty(*dst_page);
- kunmap(*dst_page);
+ kunmap_thread(*dst_page);
if (!dst)
dst_page++;
else
@@ -202,27 +202,27 @@ void hfs_bnode_move(struct hfs_bnode *node, int dst, int src, int len)
if (src == dst) {
while (src < len) {
- memmove(kmap(*dst_page), kmap(*src_page), src);
- kunmap(*src_page);
+ memmove(kmap_thread(*dst_page), kmap_thread(*src_page), src);
+ kunmap_thread(*src_page);
set_page_dirty(*dst_page);
- kunmap(*dst_page);
+ kunmap_thread(*dst_page);
len -= src;
src = PAGE_SIZE;
src_page--;
dst_page--;
}
src -= len;
- memmove(kmap(*dst_page) + src,
- kmap(*src_page) + src, len);
- kunmap(*src_page);
+ memmove(kmap_thread(*dst_page) + src,
+ kmap_thread(*src_page) + src, len);
+ kunmap_thread(*src_page);
set_page_dirty(*dst_page);
- kunmap(*dst_page);
+ kunmap_thread(*dst_page);
} else {
void *src_ptr, *dst_ptr;
do {
- src_ptr = kmap(*src_page) + src;
- dst_ptr = kmap(*dst_page) + dst;
+ src_ptr = kmap_thread(*src_page) + src;
+ dst_ptr = kmap_thread(*dst_page) + dst;
if (src < dst) {
l = src;
src = PAGE_SIZE;
@@ -234,9 +234,9 @@ void hfs_bnode_move(struct hfs_bnode *node, int dst, int src, int len)
}
l = min(len, l);
memmove(dst_ptr - l, src_ptr - l, l);
- kunmap(*src_page);
+ kunmap_thread(*src_page);
set_page_dirty(*dst_page);
- kunmap(*dst_page);
+ kunmap_thread(*dst_page);
if (dst == PAGE_SIZE)
dst_page--;
else
@@ -251,26 +251,26 @@ void hfs_bnode_move(struct hfs_bnode *node, int dst, int src, int len)
if (src == dst) {
l = min_t(int, len, PAGE_SIZE - src);
- memmove(kmap(*dst_page) + src,
- kmap(*src_page) + src, l);
- kunmap(*src_page);
+ memmove(kmap_thread(*dst_page) + src,
+ kmap_thread(*src_page) + src, l);
+ kunmap_thread(*src_page);
set_page_dirty(*dst_page);
- kunmap(*dst_page);
+ kunmap_thread(*dst_page);
while ((len -= l) != 0) {
l = min_t(int, len, PAGE_SIZE);
- memmove(kmap(*++dst_page),
- kmap(*++src_page), l);
- kunmap(*src_page);
+ memmove(kmap_thread(*++dst_page),
+ kmap_thread(*++src_page), l);
+ kunmap_thread(*src_page);
set_page_dirty(*dst_page);
- kunmap(*dst_page);
+ kunmap_thread(*dst_page);
}
} else {
void *src_ptr, *dst_ptr;
do {
- src_ptr = kmap(*src_page) + src;
- dst_ptr = kmap(*dst_page) + dst;
+ src_ptr = kmap_thread(*src_page) + src;
+ dst_ptr = kmap_thread(*dst_page) + dst;
if (PAGE_SIZE - src <
PAGE_SIZE - dst) {
l = PAGE_SIZE - src;
@@ -283,9 +283,9 @@ void hfs_bnode_move(struct hfs_bnode *node, int dst, int src, int len)
}
l = min(len, l);
memmove(dst_ptr, src_ptr, l);
- kunmap(*src_page);
+ kunmap_thread(*src_page);
set_page_dirty(*dst_page);
- kunmap(*dst_page);
+ kunmap_thread(*dst_page);
if (!dst)
dst_page++;
else
@@ -502,14 +502,14 @@ struct hfs_bnode *hfs_bnode_find(struct hfs_btree *tree, u32 num)
if (!test_bit(HFS_BNODE_NEW, &node->flags))
return node;
- desc = (struct hfs_bnode_desc *)(kmap(node->page[0]) +
+ desc = (struct hfs_bnode_desc *)(kmap_thread(node->page[0]) +
node->page_offset);
node->prev = be32_to_cpu(desc->prev);
node->next = be32_to_cpu(desc->next);
node->num_recs = be16_to_cpu(desc->num_recs);
node->type = desc->type;
node->height = desc->height;
- kunmap(node->page[0]);
+ kunmap_thread(node->page[0]);
switch (node->type) {
case HFS_NODE_HEADER:
@@ -593,14 +593,14 @@ struct hfs_bnode *hfs_bnode_create(struct hfs_btree *tree, u32 num)
}
pagep = node->page;
- memset(kmap(*pagep) + node->page_offset, 0,
+ memset(kmap_thread(*pagep) + node->page_offset, 0,
min_t(int, PAGE_SIZE, tree->node_size));
set_page_dirty(*pagep);
- kunmap(*pagep);
+ kunmap_thread(*pagep);
for (i = 1; i < tree->pages_per_bnode; i++) {
- memset(kmap(*++pagep), 0, PAGE_SIZE);
+ memset(kmap_thread(*++pagep), 0, PAGE_SIZE);
set_page_dirty(*pagep);
- kunmap(*pagep);
+ kunmap_thread(*pagep);
}
clear_bit(HFS_BNODE_NEW, &node->flags);
wake_up(&node->lock_wq);
diff --git a/fs/hfsplus/btree.c b/fs/hfsplus/btree.c
index 66774f4cb4fd..74fcef3a1628 100644
--- a/fs/hfsplus/btree.c
+++ b/fs/hfsplus/btree.c
@@ -394,7 +394,7 @@ struct hfs_bnode *hfs_bmap_alloc(struct hfs_btree *tree)
off += node->page_offset;
pagep = node->page + (off >> PAGE_SHIFT);
- data = kmap(*pagep);
+ data = kmap_thread(*pagep);
off &= ~PAGE_MASK;
idx = 0;
@@ -407,7 +407,7 @@ struct hfs_bnode *hfs_bmap_alloc(struct hfs_btree *tree)
idx += i;
data[off] |= m;
set_page_dirty(*pagep);
- kunmap(*pagep);
+ kunmap_thread(*pagep);
tree->free_nodes--;
mark_inode_dirty(tree->inode);
hfs_bnode_put(node);
@@ -417,14 +417,14 @@ struct hfs_bnode *hfs_bmap_alloc(struct hfs_btree *tree)
}
}
if (++off >= PAGE_SIZE) {
- kunmap(*pagep);
- data = kmap(*++pagep);
+ kunmap_thread(*pagep);
+ data = kmap_thread(*++pagep);
off = 0;
}
idx += 8;
len--;
}
- kunmap(*pagep);
+ kunmap_thread(*pagep);
nidx = node->next;
if (!nidx) {
hfs_dbg(BNODE_MOD, "create new bmap node\n");
@@ -440,7 +440,7 @@ struct hfs_bnode *hfs_bmap_alloc(struct hfs_btree *tree)
off = off16;
off += node->page_offset;
pagep = node->page + (off >> PAGE_SHIFT);
- data = kmap(*pagep);
+ data = kmap_thread(*pagep);
off &= ~PAGE_MASK;
}
}
@@ -490,7 +490,7 @@ void hfs_bmap_free(struct hfs_bnode *node)
}
off += node->page_offset + nidx / 8;
page = node->page[off >> PAGE_SHIFT];
- data = kmap(page);
+ data = kmap_thread(page);
off &= ~PAGE_MASK;
m = 1 << (~nidx & 7);
byte = data[off];
@@ -498,13 +498,13 @@ void hfs_bmap_free(struct hfs_bnode *node)
pr_crit("trying to free free bnode "
"%u(%d)\n",
node->this, node->type);
- kunmap(page);
+ kunmap_thread(page);
hfs_bnode_put(node);
return;
}
data[off] = byte & ~m;
set_page_dirty(page);
- kunmap(page);
+ kunmap_thread(page);
hfs_bnode_put(node);
tree->free_nodes++;
mark_inode_dirty(tree->inode);
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 18/58] fs/hfs: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-um,
intel-gfx, ecryptfs, linux-erofs, reiserfs-devel, linux-block,
linux-bcache, Dan Williams, io-uring, linux-nfs, linux-ntfs-dev,
netdev, kexec, linux-kernel, linux-f2fs-devel, linux-fsdevel, bpf,
linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this FS are localized to a single thread. To avoid
the over head of global PKRS updates use the new kmap_thread() call.
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
fs/hfs/bnode.c | 14 +++++++-------
fs/hfs/btree.c | 20 ++++++++++----------
2 files changed, 17 insertions(+), 17 deletions(-)
diff --git a/fs/hfs/bnode.c b/fs/hfs/bnode.c
index b63a4df7327b..8b4d02576405 100644
--- a/fs/hfs/bnode.c
+++ b/fs/hfs/bnode.c
@@ -23,8 +23,8 @@ void hfs_bnode_read(struct hfs_bnode *node, void *buf,
off += node->page_offset;
page = node->page[0];
- memcpy(buf, kmap(page) + off, len);
- kunmap(page);
+ memcpy(buf, kmap_thread(page) + off, len);
+ kunmap_thread(page);
}
u16 hfs_bnode_read_u16(struct hfs_bnode *node, int off)
@@ -108,9 +108,9 @@ void hfs_bnode_copy(struct hfs_bnode *dst_node, int dst,
src_page = src_node->page[0];
dst_page = dst_node->page[0];
- memcpy(kmap(dst_page) + dst, kmap(src_page) + src, len);
- kunmap(src_page);
- kunmap(dst_page);
+ memcpy(kmap_thread(dst_page) + dst, kmap_thread(src_page) + src, len);
+ kunmap_thread(src_page);
+ kunmap_thread(dst_page);
set_page_dirty(dst_page);
}
@@ -125,9 +125,9 @@ void hfs_bnode_move(struct hfs_bnode *node, int dst, int src, int len)
src += node->page_offset;
dst += node->page_offset;
page = node->page[0];
- ptr = kmap(page);
+ ptr = kmap_thread(page);
memmove(ptr + dst, ptr + src, len);
- kunmap(page);
+ kunmap_thread(page);
set_page_dirty(page);
}
diff --git a/fs/hfs/btree.c b/fs/hfs/btree.c
index 19017d296173..bd4a6d35e361 100644
--- a/fs/hfs/btree.c
+++ b/fs/hfs/btree.c
@@ -80,7 +80,7 @@ struct hfs_btree *hfs_btree_open(struct super_block *sb, u32 id, btree_keycmp ke
goto free_inode;
/* Load the header */
- head = (struct hfs_btree_header_rec *)(kmap(page) + sizeof(struct hfs_bnode_desc));
+ head = (struct hfs_btree_header_rec *)(kmap_thread(page) + sizeof(struct hfs_bnode_desc));
tree->root = be32_to_cpu(head->root);
tree->leaf_count = be32_to_cpu(head->leaf_count);
tree->leaf_head = be32_to_cpu(head->leaf_head);
@@ -119,7 +119,7 @@ struct hfs_btree *hfs_btree_open(struct super_block *sb, u32 id, btree_keycmp ke
tree->node_size_shift = ffs(size) - 1;
tree->pages_per_bnode = (tree->node_size + PAGE_SIZE - 1) >> PAGE_SHIFT;
- kunmap(page);
+ kunmap_thread(page);
put_page(page);
return tree;
@@ -268,7 +268,7 @@ struct hfs_bnode *hfs_bmap_alloc(struct hfs_btree *tree)
off += node->page_offset;
pagep = node->page + (off >> PAGE_SHIFT);
- data = kmap(*pagep);
+ data = kmap_thread(*pagep);
off &= ~PAGE_MASK;
idx = 0;
@@ -281,7 +281,7 @@ struct hfs_bnode *hfs_bmap_alloc(struct hfs_btree *tree)
idx += i;
data[off] |= m;
set_page_dirty(*pagep);
- kunmap(*pagep);
+ kunmap_thread(*pagep);
tree->free_nodes--;
mark_inode_dirty(tree->inode);
hfs_bnode_put(node);
@@ -290,14 +290,14 @@ struct hfs_bnode *hfs_bmap_alloc(struct hfs_btree *tree)
}
}
if (++off >= PAGE_SIZE) {
- kunmap(*pagep);
- data = kmap(*++pagep);
+ kunmap_thread(*pagep);
+ data = kmap_thread(*++pagep);
off = 0;
}
idx += 8;
len--;
}
- kunmap(*pagep);
+ kunmap_thread(*pagep);
nidx = node->next;
if (!nidx) {
printk(KERN_DEBUG "create new bmap node...\n");
@@ -313,7 +313,7 @@ struct hfs_bnode *hfs_bmap_alloc(struct hfs_btree *tree)
off = off16;
off += node->page_offset;
pagep = node->page + (off >> PAGE_SHIFT);
- data = kmap(*pagep);
+ data = kmap_thread(*pagep);
off &= ~PAGE_MASK;
}
}
@@ -360,7 +360,7 @@ void hfs_bmap_free(struct hfs_bnode *node)
}
off += node->page_offset + nidx / 8;
page = node->page[off >> PAGE_SHIFT];
- data = kmap(page);
+ data = kmap_thread(page);
off &= ~PAGE_MASK;
m = 1 << (~nidx & 7);
byte = data[off];
@@ -373,7 +373,7 @@ void hfs_bmap_free(struct hfs_bnode *node)
}
data[off] = byte & ~m;
set_page_dirty(page);
- kunmap(page);
+ kunmap_thread(page);
hfs_bnode_put(node);
tree->free_nodes++;
mark_inode_dirty(tree->inode);
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 17/58] fs/nilfs2: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, io-uring, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-afs,
linux-um, intel-gfx, ecryptfs, linux-erofs, reiserfs-devel,
linux-block, linux-bcache, Dan Williams, Ryusuke Konishi,
linux-nfs, linux-ntfs-dev, netdev, kexec, linux-kernel,
linux-f2fs-devel, linux-fsdevel, bpf, linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this FS are localized to a single thread. To avoid
the over head of global PKRS updates use the new kmap_thread() call.
Cc: Ryusuke Konishi <konishi.ryusuke@gmail.com>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
fs/nilfs2/alloc.c | 34 +++++++++++++++++-----------------
fs/nilfs2/cpfile.c | 4 ++--
2 files changed, 19 insertions(+), 19 deletions(-)
diff --git a/fs/nilfs2/alloc.c b/fs/nilfs2/alloc.c
index adf3bb0a8048..2aa4c34094ef 100644
--- a/fs/nilfs2/alloc.c
+++ b/fs/nilfs2/alloc.c
@@ -524,7 +524,7 @@ int nilfs_palloc_prepare_alloc_entry(struct inode *inode,
ret = nilfs_palloc_get_desc_block(inode, group, 1, &desc_bh);
if (ret < 0)
return ret;
- desc_kaddr = kmap(desc_bh->b_page);
+ desc_kaddr = kmap_thread(desc_bh->b_page);
desc = nilfs_palloc_block_get_group_desc(
inode, group, desc_bh, desc_kaddr);
n = nilfs_palloc_rest_groups_in_desc_block(inode, group,
@@ -536,7 +536,7 @@ int nilfs_palloc_prepare_alloc_entry(struct inode *inode,
inode, group, 1, &bitmap_bh);
if (ret < 0)
goto out_desc;
- bitmap_kaddr = kmap(bitmap_bh->b_page);
+ bitmap_kaddr = kmap_thread(bitmap_bh->b_page);
bitmap = bitmap_kaddr + bh_offset(bitmap_bh);
pos = nilfs_palloc_find_available_slot(
bitmap, group_offset,
@@ -547,21 +547,21 @@ int nilfs_palloc_prepare_alloc_entry(struct inode *inode,
desc, lock, -1);
req->pr_entry_nr =
entries_per_group * group + pos;
- kunmap(desc_bh->b_page);
- kunmap(bitmap_bh->b_page);
+ kunmap_thread(desc_bh->b_page);
+ kunmap_thread(bitmap_bh->b_page);
req->pr_desc_bh = desc_bh;
req->pr_bitmap_bh = bitmap_bh;
return 0;
}
- kunmap(bitmap_bh->b_page);
+ kunmap_thread(bitmap_bh->b_page);
brelse(bitmap_bh);
}
group_offset = 0;
}
- kunmap(desc_bh->b_page);
+ kunmap_thread(desc_bh->b_page);
brelse(desc_bh);
}
@@ -569,7 +569,7 @@ int nilfs_palloc_prepare_alloc_entry(struct inode *inode,
return -ENOSPC;
out_desc:
- kunmap(desc_bh->b_page);
+ kunmap_thread(desc_bh->b_page);
brelse(desc_bh);
return ret;
}
@@ -605,10 +605,10 @@ void nilfs_palloc_commit_free_entry(struct inode *inode,
spinlock_t *lock;
group = nilfs_palloc_group(inode, req->pr_entry_nr, &group_offset);
- desc_kaddr = kmap(req->pr_desc_bh->b_page);
+ desc_kaddr = kmap_thread(req->pr_desc_bh->b_page);
desc = nilfs_palloc_block_get_group_desc(inode, group,
req->pr_desc_bh, desc_kaddr);
- bitmap_kaddr = kmap(req->pr_bitmap_bh->b_page);
+ bitmap_kaddr = kmap_thread(req->pr_bitmap_bh->b_page);
bitmap = bitmap_kaddr + bh_offset(req->pr_bitmap_bh);
lock = nilfs_mdt_bgl_lock(inode, group);
@@ -620,8 +620,8 @@ void nilfs_palloc_commit_free_entry(struct inode *inode,
else
nilfs_palloc_group_desc_add_entries(desc, lock, 1);
- kunmap(req->pr_bitmap_bh->b_page);
- kunmap(req->pr_desc_bh->b_page);
+ kunmap_thread(req->pr_bitmap_bh->b_page);
+ kunmap_thread(req->pr_desc_bh->b_page);
mark_buffer_dirty(req->pr_desc_bh);
mark_buffer_dirty(req->pr_bitmap_bh);
@@ -646,10 +646,10 @@ void nilfs_palloc_abort_alloc_entry(struct inode *inode,
spinlock_t *lock;
group = nilfs_palloc_group(inode, req->pr_entry_nr, &group_offset);
- desc_kaddr = kmap(req->pr_desc_bh->b_page);
+ desc_kaddr = kmap_thread(req->pr_desc_bh->b_page);
desc = nilfs_palloc_block_get_group_desc(inode, group,
req->pr_desc_bh, desc_kaddr);
- bitmap_kaddr = kmap(req->pr_bitmap_bh->b_page);
+ bitmap_kaddr = kmap_thread(req->pr_bitmap_bh->b_page);
bitmap = bitmap_kaddr + bh_offset(req->pr_bitmap_bh);
lock = nilfs_mdt_bgl_lock(inode, group);
@@ -661,8 +661,8 @@ void nilfs_palloc_abort_alloc_entry(struct inode *inode,
else
nilfs_palloc_group_desc_add_entries(desc, lock, 1);
- kunmap(req->pr_bitmap_bh->b_page);
- kunmap(req->pr_desc_bh->b_page);
+ kunmap_thread(req->pr_bitmap_bh->b_page);
+ kunmap_thread(req->pr_desc_bh->b_page);
brelse(req->pr_bitmap_bh);
brelse(req->pr_desc_bh);
@@ -754,7 +754,7 @@ int nilfs_palloc_freev(struct inode *inode, __u64 *entry_nrs, size_t nitems)
/* Get the first entry number of the group */
group_min_nr = (__u64)group * epg;
- bitmap_kaddr = kmap(bitmap_bh->b_page);
+ bitmap_kaddr = kmap_thread(bitmap_bh->b_page);
bitmap = bitmap_kaddr + bh_offset(bitmap_bh);
lock = nilfs_mdt_bgl_lock(inode, group);
@@ -800,7 +800,7 @@ int nilfs_palloc_freev(struct inode *inode, __u64 *entry_nrs, size_t nitems)
entry_start = rounddown(group_offset, epb);
} while (true);
- kunmap(bitmap_bh->b_page);
+ kunmap_thread(bitmap_bh->b_page);
mark_buffer_dirty(bitmap_bh);
brelse(bitmap_bh);
diff --git a/fs/nilfs2/cpfile.c b/fs/nilfs2/cpfile.c
index 86d4d850d130..402ab8bfce29 100644
--- a/fs/nilfs2/cpfile.c
+++ b/fs/nilfs2/cpfile.c
@@ -235,11 +235,11 @@ int nilfs_cpfile_get_checkpoint(struct inode *cpfile,
ret = nilfs_cpfile_get_checkpoint_block(cpfile, cno, create, &cp_bh);
if (ret < 0)
goto out_header;
- kaddr = kmap(cp_bh->b_page);
+ kaddr = kmap_thread(cp_bh->b_page);
cp = nilfs_cpfile_block_get_checkpoint(cpfile, cno, cp_bh, kaddr);
if (nilfs_checkpoint_invalid(cp)) {
if (!create) {
- kunmap(cp_bh->b_page);
+ kunmap_thread(cp_bh->b_page);
brelse(cp_bh);
ret = -ENOENT;
goto out_header;
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 16/58] fs/gfs2: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, Andreas Gruenbacher, linux-scsi,
linux-nvdimm, linux-rdma, x86, amd-gfx, linux-afs, cluster-devel,
linux-cachefs, intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu,
linux-um, intel-gfx, ecryptfs, linux-erofs, reiserfs-devel,
linux-block, linux-fsdevel, linux-bcache, Dan Williams, io-uring,
linux-nfs, linux-ntfs-dev, netdev, kexec, linux-kernel,
linux-f2fs-devel, Bob Peterson, bpf, linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this FS are localized to a single thread. To avoid
the over head of global PKRS updates use the new kmap_thread() call.
Cc: Bob Peterson <rpeterso@redhat.com>
Cc: Andreas Gruenbacher <agruenba@redhat.com>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
fs/gfs2/bmap.c | 4 ++--
fs/gfs2/ops_fstype.c | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c
index 0f69fbd4af66..375af4528411 100644
--- a/fs/gfs2/bmap.c
+++ b/fs/gfs2/bmap.c
@@ -67,7 +67,7 @@ static int gfs2_unstuffer_page(struct gfs2_inode *ip, struct buffer_head *dibh,
}
if (!PageUptodate(page)) {
- void *kaddr = kmap(page);
+ void *kaddr = kmap_thread(page);
u64 dsize = i_size_read(inode);
if (dsize > gfs2_max_stuffed_size(ip))
@@ -75,7 +75,7 @@ static int gfs2_unstuffer_page(struct gfs2_inode *ip, struct buffer_head *dibh,
memcpy(kaddr, dibh->b_data + sizeof(struct gfs2_dinode), dsize);
memset(kaddr + dsize, 0, PAGE_SIZE - dsize);
- kunmap(page);
+ kunmap_thread(page);
SetPageUptodate(page);
}
diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c
index 6d18d2c91add..a5d20d9b504a 100644
--- a/fs/gfs2/ops_fstype.c
+++ b/fs/gfs2/ops_fstype.c
@@ -263,9 +263,9 @@ static int gfs2_read_super(struct gfs2_sbd *sdp, sector_t sector, int silent)
__free_page(page);
return -EIO;
}
- p = kmap(page);
+ p = kmap_thread(page);
gfs2_sb_in(sdp, p);
- kunmap(page);
+ kunmap_thread(page);
__free_page(page);
return gfs2_check_sb(sdp, silent);
}
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 15/58] fs/ecryptfs: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, Eric Biggers, kexec, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, Herbert Xu, linux-scsi, linux-nvdimm,
linux-rdma, x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, Aditya Pakki, linux-mmc, linux-ext4, Fenghua Yu,
linux-um, intel-gfx, ecryptfs, linux-erofs, reiserfs-devel,
linux-block, linux-bcache, xen-devel, Dan Williams, io-uring,
linux-nfs, linux-ntfs-dev, netdev, linux-doc, linux-kernel,
linux-f2fs-devel, linux-fsdevel, bpf, linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this FS are localized to a single thread. To avoid
the over head of global PKRS updates use the new kmap_thread() call.
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Eric Biggers <ebiggers@google.com>
Cc: Aditya Pakki <pakki001@umn.edu>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
fs/ecryptfs/crypto.c | 8 ++++----
fs/ecryptfs/read_write.c | 8 ++++----
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/fs/ecryptfs/crypto.c b/fs/ecryptfs/crypto.c
index 0681540c48d9..e73e00994bee 100644
--- a/fs/ecryptfs/crypto.c
+++ b/fs/ecryptfs/crypto.c
@@ -469,10 +469,10 @@ int ecryptfs_encrypt_page(struct page *page)
}
lower_offset = lower_offset_for_page(crypt_stat, page);
- enc_extent_virt = kmap(enc_extent_page);
+ enc_extent_virt = kmap_thread(enc_extent_page);
rc = ecryptfs_write_lower(ecryptfs_inode, enc_extent_virt, lower_offset,
PAGE_SIZE);
- kunmap(enc_extent_page);
+ kunmap_thread(enc_extent_page);
if (rc < 0) {
ecryptfs_printk(KERN_ERR,
"Error attempting to write lower page; rc = [%d]\n",
@@ -518,10 +518,10 @@ int ecryptfs_decrypt_page(struct page *page)
BUG_ON(!(crypt_stat->flags & ECRYPTFS_ENCRYPTED));
lower_offset = lower_offset_for_page(crypt_stat, page);
- page_virt = kmap(page);
+ page_virt = kmap_thread(page);
rc = ecryptfs_read_lower(page_virt, lower_offset, PAGE_SIZE,
ecryptfs_inode);
- kunmap(page);
+ kunmap_thread(page);
if (rc < 0) {
ecryptfs_printk(KERN_ERR,
"Error attempting to read lower page; rc = [%d]\n",
diff --git a/fs/ecryptfs/read_write.c b/fs/ecryptfs/read_write.c
index 0438997ac9d8..5eca4330c0c0 100644
--- a/fs/ecryptfs/read_write.c
+++ b/fs/ecryptfs/read_write.c
@@ -64,11 +64,11 @@ int ecryptfs_write_lower_page_segment(struct inode *ecryptfs_inode,
offset = ((((loff_t)page_for_lower->index) << PAGE_SHIFT)
+ offset_in_page);
- virt = kmap(page_for_lower);
+ virt = kmap_thread(page_for_lower);
rc = ecryptfs_write_lower(ecryptfs_inode, virt, offset, size);
if (rc > 0)
rc = 0;
- kunmap(page_for_lower);
+ kunmap_thread(page_for_lower);
return rc;
}
@@ -251,11 +251,11 @@ int ecryptfs_read_lower_page_segment(struct page *page_for_ecryptfs,
int rc;
offset = ((((loff_t)page_index) << PAGE_SHIFT) + offset_in_page);
- virt = kmap(page_for_ecryptfs);
+ virt = kmap_thread(page_for_ecryptfs);
rc = ecryptfs_read_lower(virt, offset, size, ecryptfs_inode);
if (rc > 0)
rc = 0;
- kunmap(page_for_ecryptfs);
+ kunmap_thread(page_for_ecryptfs);
flush_dcache_page(page_for_ecryptfs);
return rc;
}
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 14/58] fs/cifs: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-um,
intel-gfx, ecryptfs, linux-erofs, reiserfs-devel, linux-block,
linux-bcache, Dan Williams, io-uring, linux-nfs, linux-ntfs-dev,
netdev, kexec, linux-kernel, linux-f2fs-devel, Steve French,
linux-fsdevel, bpf, linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this FS are localized to a single thread. To avoid
the over head of global PKRS updates use the new kmap_thread() call.
Cc: Steve French <sfrench@samba.org>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
fs/cifs/cifsencrypt.c | 6 +++---
fs/cifs/file.c | 16 ++++++++--------
fs/cifs/smb2ops.c | 8 ++++----
3 files changed, 15 insertions(+), 15 deletions(-)
diff --git a/fs/cifs/cifsencrypt.c b/fs/cifs/cifsencrypt.c
index 9daa256f69d4..2f8232d01a56 100644
--- a/fs/cifs/cifsencrypt.c
+++ b/fs/cifs/cifsencrypt.c
@@ -82,17 +82,17 @@ int __cifs_calc_signature(struct smb_rqst *rqst,
rqst_page_get_length(rqst, i, &len, &offset);
- kaddr = (char *) kmap(rqst->rq_pages[i]) + offset;
+ kaddr = (char *) kmap_thread(rqst->rq_pages[i]) + offset;
rc = crypto_shash_update(shash, kaddr, len);
if (rc) {
cifs_dbg(VFS, "%s: Could not update with payload\n",
__func__);
- kunmap(rqst->rq_pages[i]);
+ kunmap_thread(rqst->rq_pages[i]);
return rc;
}
- kunmap(rqst->rq_pages[i]);
+ kunmap_thread(rqst->rq_pages[i]);
}
rc = crypto_shash_final(shash, signature);
diff --git a/fs/cifs/file.c b/fs/cifs/file.c
index be46fab4c96d..6db2caab8852 100644
--- a/fs/cifs/file.c
+++ b/fs/cifs/file.c
@@ -2145,17 +2145,17 @@ static int cifs_partialpagewrite(struct page *page, unsigned from, unsigned to)
inode = page->mapping->host;
offset += (loff_t)from;
- write_data = kmap(page);
+ write_data = kmap_thread(page);
write_data += from;
if ((to > PAGE_SIZE) || (from > to)) {
- kunmap(page);
+ kunmap_thread(page);
return -EIO;
}
/* racing with truncate? */
if (offset > mapping->host->i_size) {
- kunmap(page);
+ kunmap_thread(page);
return 0; /* don't care */
}
@@ -2183,7 +2183,7 @@ static int cifs_partialpagewrite(struct page *page, unsigned from, unsigned to)
rc = -EIO;
}
- kunmap(page);
+ kunmap_thread(page);
return rc;
}
@@ -2559,10 +2559,10 @@ static int cifs_write_end(struct file *file, struct address_space *mapping,
known which we might as well leverage */
/* BB check if anything else missing out of ppw
such as updating last write time */
- page_data = kmap(page);
+ page_data = kmap_thread(page);
rc = cifs_write(cfile, pid, page_data + offset, copied, &pos);
/* if (rc < 0) should we set writebehind rc? */
- kunmap(page);
+ kunmap_thread(page);
free_xid(xid);
} else {
@@ -4511,7 +4511,7 @@ static int cifs_readpage_worker(struct file *file, struct page *page,
if (rc == 0)
goto read_complete;
- read_data = kmap(page);
+ read_data = kmap_thread(page);
/* for reads over a certain size could initiate async read ahead */
rc = cifs_read(file, read_data, PAGE_SIZE, poffset);
@@ -4540,7 +4540,7 @@ static int cifs_readpage_worker(struct file *file, struct page *page,
rc = 0;
io_error:
- kunmap(page);
+ kunmap_thread(page);
unlock_page(page);
read_complete:
diff --git a/fs/cifs/smb2ops.c b/fs/cifs/smb2ops.c
index 32f90dc82c84..a3e7ebab38b6 100644
--- a/fs/cifs/smb2ops.c
+++ b/fs/cifs/smb2ops.c
@@ -4068,12 +4068,12 @@ smb3_init_transform_rq(struct TCP_Server_Info *server, int num_rqst,
rqst_page_get_length(&new_rq[i], j, &len, &offset);
- dst = (char *) kmap(new_rq[i].rq_pages[j]) + offset;
- src = (char *) kmap(old_rq[i - 1].rq_pages[j]) + offset;
+ dst = (char *) kmap_thread(new_rq[i].rq_pages[j]) + offset;
+ src = (char *) kmap_thread(old_rq[i - 1].rq_pages[j]) + offset;
memcpy(dst, src, len);
- kunmap(new_rq[i].rq_pages[j]);
- kunmap(old_rq[i - 1].rq_pages[j]);
+ kunmap_thread(new_rq[i].rq_pages[j]);
+ kunmap_thread(old_rq[i - 1].rq_pages[j]);
}
}
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 13/58] fs/btrfs: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-um,
intel-gfx, ecryptfs, linux-erofs, Josef Bacik, reiserfs-devel,
linux-block, linux-bcache, David Sterba, Dan Williams, io-uring,
linux-nfs, Chris Mason, linux-ntfs-dev, netdev, kexec,
linux-kernel, linux-f2fs-devel, linux-fsdevel, bpf, linuxppc-dev,
linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this FS are localized to a single thread. To avoid
the over head of global PKRS updates use the new kmap_thread() call.
Cc: Chris Mason <clm@fb.com>
Cc: Josef Bacik <josef@toxicpanda.com>
Cc: David Sterba <dsterba@suse.com>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
fs/btrfs/check-integrity.c | 4 ++--
fs/btrfs/compression.c | 4 ++--
fs/btrfs/inode.c | 16 ++++++++--------
fs/btrfs/lzo.c | 24 ++++++++++++------------
fs/btrfs/raid56.c | 34 +++++++++++++++++-----------------
fs/btrfs/reflink.c | 8 ++++----
fs/btrfs/send.c | 4 ++--
fs/btrfs/zlib.c | 32 ++++++++++++++++----------------
fs/btrfs/zstd.c | 20 ++++++++++----------
9 files changed, 73 insertions(+), 73 deletions(-)
diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c
index 81a8c87a5afb..9e5a02512ab5 100644
--- a/fs/btrfs/check-integrity.c
+++ b/fs/btrfs/check-integrity.c
@@ -2706,7 +2706,7 @@ static void __btrfsic_submit_bio(struct bio *bio)
bio_for_each_segment(bvec, bio, iter) {
BUG_ON(bvec.bv_len != PAGE_SIZE);
- mapped_datav[i] = kmap(bvec.bv_page);
+ mapped_datav[i] = kmap_thread(bvec.bv_page);
i++;
if (dev_state->state->print_mask &
@@ -2720,7 +2720,7 @@ static void __btrfsic_submit_bio(struct bio *bio)
bio, &bio_is_patched,
bio->bi_opf);
bio_for_each_segment(bvec, bio, iter)
- kunmap(bvec.bv_page);
+ kunmap_thread(bvec.bv_page);
kfree(mapped_datav);
} else if (NULL != dev_state && (bio->bi_opf & REQ_PREFLUSH)) {
if (dev_state->state->print_mask &
diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c
index 1ab56a734e70..5944fb36d68a 100644
--- a/fs/btrfs/compression.c
+++ b/fs/btrfs/compression.c
@@ -1626,7 +1626,7 @@ static void heuristic_collect_sample(struct inode *inode, u64 start, u64 end,
curr_sample_pos = 0;
while (index < index_end) {
page = find_get_page(inode->i_mapping, index);
- in_data = kmap(page);
+ in_data = kmap_thread(page);
/* Handle case where the start is not aligned to PAGE_SIZE */
i = start % PAGE_SIZE;
while (i < PAGE_SIZE - SAMPLING_READ_SIZE) {
@@ -1639,7 +1639,7 @@ static void heuristic_collect_sample(struct inode *inode, u64 start, u64 end,
start += SAMPLING_INTERVAL;
curr_sample_pos += SAMPLING_READ_SIZE;
}
- kunmap(page);
+ kunmap_thread(page);
put_page(page);
index++;
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 9570458aa847..9710a52c6c42 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -4603,7 +4603,7 @@ int btrfs_truncate_block(struct inode *inode, loff_t from, loff_t len,
if (offset != blocksize) {
if (!len)
len = blocksize - offset;
- kaddr = kmap(page);
+ kaddr = kmap_thread(page);
if (front)
memset(kaddr + (block_start - page_offset(page)),
0, offset);
@@ -4611,7 +4611,7 @@ int btrfs_truncate_block(struct inode *inode, loff_t from, loff_t len,
memset(kaddr + (block_start - page_offset(page)) + offset,
0, len);
flush_dcache_page(page);
- kunmap(page);
+ kunmap_thread(page);
}
ClearPageChecked(page);
set_page_dirty(page);
@@ -6509,9 +6509,9 @@ static noinline int uncompress_inline(struct btrfs_path *path,
*/
if (max_size + pg_offset < PAGE_SIZE) {
- char *map = kmap(page);
+ char *map = kmap_thread(page);
memset(map + pg_offset + max_size, 0, PAGE_SIZE - max_size - pg_offset);
- kunmap(page);
+ kunmap_thread(page);
}
kfree(tmp);
return ret;
@@ -6704,7 +6704,7 @@ struct extent_map *btrfs_get_extent(struct btrfs_inode *inode,
goto out;
}
} else {
- map = kmap(page);
+ map = kmap_thread(page);
read_extent_buffer(leaf, map + pg_offset, ptr,
copy_size);
if (pg_offset + copy_size < PAGE_SIZE) {
@@ -6712,7 +6712,7 @@ struct extent_map *btrfs_get_extent(struct btrfs_inode *inode,
PAGE_SIZE - pg_offset -
copy_size);
}
- kunmap(page);
+ kunmap_thread(page);
}
flush_dcache_page(page);
}
@@ -8326,10 +8326,10 @@ vm_fault_t btrfs_page_mkwrite(struct vm_fault *vmf)
zero_start = PAGE_SIZE;
if (zero_start != PAGE_SIZE) {
- kaddr = kmap(page);
+ kaddr = kmap_thread(page);
memset(kaddr + zero_start, 0, PAGE_SIZE - zero_start);
flush_dcache_page(page);
- kunmap(page);
+ kunmap_thread(page);
}
ClearPageChecked(page);
set_page_dirty(page);
diff --git a/fs/btrfs/lzo.c b/fs/btrfs/lzo.c
index aa9cd11f4b78..f29dcc9ec573 100644
--- a/fs/btrfs/lzo.c
+++ b/fs/btrfs/lzo.c
@@ -140,7 +140,7 @@ int lzo_compress_pages(struct list_head *ws, struct address_space *mapping,
*total_in = 0;
in_page = find_get_page(mapping, start >> PAGE_SHIFT);
- data_in = kmap(in_page);
+ data_in = kmap_thread(in_page);
/*
* store the size of all chunks of compressed data in
@@ -151,7 +151,7 @@ int lzo_compress_pages(struct list_head *ws, struct address_space *mapping,
ret = -ENOMEM;
goto out;
}
- cpage_out = kmap(out_page);
+ cpage_out = kmap_thread(out_page);
out_offset = LZO_LEN;
tot_out = LZO_LEN;
pages[0] = out_page;
@@ -209,7 +209,7 @@ int lzo_compress_pages(struct list_head *ws, struct address_space *mapping,
if (out_len == 0 && tot_in >= len)
break;
- kunmap(out_page);
+ kunmap_thread(out_page);
if (nr_pages == nr_dest_pages) {
out_page = NULL;
ret = -E2BIG;
@@ -221,7 +221,7 @@ int lzo_compress_pages(struct list_head *ws, struct address_space *mapping,
ret = -ENOMEM;
goto out;
}
- cpage_out = kmap(out_page);
+ cpage_out = kmap_thread(out_page);
pages[nr_pages++] = out_page;
pg_bytes_left = PAGE_SIZE;
@@ -243,12 +243,12 @@ int lzo_compress_pages(struct list_head *ws, struct address_space *mapping,
break;
bytes_left = len - tot_in;
- kunmap(in_page);
+ kunmap_thread(in_page);
put_page(in_page);
start += PAGE_SIZE;
in_page = find_get_page(mapping, start >> PAGE_SHIFT);
- data_in = kmap(in_page);
+ data_in = kmap_thread(in_page);
in_len = min(bytes_left, PAGE_SIZE);
}
@@ -258,10 +258,10 @@ int lzo_compress_pages(struct list_head *ws, struct address_space *mapping,
}
/* store the size of all chunks of compressed data */
- cpage_out = kmap(pages[0]);
+ cpage_out = kmap_thread(pages[0]);
write_compress_length(cpage_out, tot_out);
- kunmap(pages[0]);
+ kunmap_thread(pages[0]);
ret = 0;
*total_out = tot_out;
@@ -269,10 +269,10 @@ int lzo_compress_pages(struct list_head *ws, struct address_space *mapping,
out:
*out_pages = nr_pages;
if (out_page)
- kunmap(out_page);
+ kunmap_thread(out_page);
if (in_page) {
- kunmap(in_page);
+ kunmap_thread(in_page);
put_page(in_page);
}
@@ -305,7 +305,7 @@ int lzo_decompress_bio(struct list_head *ws, struct compressed_bio *cb)
u64 disk_start = cb->start;
struct bio *orig_bio = cb->orig_bio;
- data_in = kmap(pages_in[0]);
+ data_in = kmap_thread(pages_in[0]);
tot_len = read_compress_length(data_in);
/*
* Compressed data header check.
@@ -387,7 +387,7 @@ int lzo_decompress_bio(struct list_head *ws, struct compressed_bio *cb)
else
kunmap(pages_in[page_in_index]);
- data_in = kmap(pages_in[++page_in_index]);
+ data_in = kmap_thread(pages_in[++page_in_index]);
in_page_bytes_left = PAGE_SIZE;
in_offset = 0;
diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c
index 255490f42b5d..34e646e4548c 100644
--- a/fs/btrfs/raid56.c
+++ b/fs/btrfs/raid56.c
@@ -262,13 +262,13 @@ static void cache_rbio_pages(struct btrfs_raid_bio *rbio)
if (!rbio->bio_pages[i])
continue;
- s = kmap(rbio->bio_pages[i]);
- d = kmap(rbio->stripe_pages[i]);
+ s = kmap_thread(rbio->bio_pages[i]);
+ d = kmap_thread(rbio->stripe_pages[i]);
copy_page(d, s);
- kunmap(rbio->bio_pages[i]);
- kunmap(rbio->stripe_pages[i]);
+ kunmap_thread(rbio->bio_pages[i]);
+ kunmap_thread(rbio->stripe_pages[i]);
SetPageUptodate(rbio->stripe_pages[i]);
}
set_bit(RBIO_CACHE_READY_BIT, &rbio->flags);
@@ -1241,13 +1241,13 @@ static noinline void finish_rmw(struct btrfs_raid_bio *rbio)
/* first collect one page from each data stripe */
for (stripe = 0; stripe < nr_data; stripe++) {
p = page_in_rbio(rbio, stripe, pagenr, 0);
- pointers[stripe] = kmap(p);
+ pointers[stripe] = kmap_thread(p);
}
/* then add the parity stripe */
p = rbio_pstripe_page(rbio, pagenr);
SetPageUptodate(p);
- pointers[stripe++] = kmap(p);
+ pointers[stripe++] = kmap_thread(p);
if (has_qstripe) {
@@ -1257,7 +1257,7 @@ static noinline void finish_rmw(struct btrfs_raid_bio *rbio)
*/
p = rbio_qstripe_page(rbio, pagenr);
SetPageUptodate(p);
- pointers[stripe++] = kmap(p);
+ pointers[stripe++] = kmap_thread(p);
raid6_call.gen_syndrome(rbio->real_stripes, PAGE_SIZE,
pointers);
@@ -1269,7 +1269,7 @@ static noinline void finish_rmw(struct btrfs_raid_bio *rbio)
for (stripe = 0; stripe < rbio->real_stripes; stripe++)
- kunmap(page_in_rbio(rbio, stripe, pagenr, 0));
+ kunmap_thread(page_in_rbio(rbio, stripe, pagenr, 0));
}
/*
@@ -1835,7 +1835,7 @@ static void __raid_recover_end_io(struct btrfs_raid_bio *rbio)
} else {
page = rbio_stripe_page(rbio, stripe, pagenr);
}
- pointers[stripe] = kmap(page);
+ pointers[stripe] = kmap_thread(page);
}
/* all raid6 handling here */
@@ -1940,7 +1940,7 @@ static void __raid_recover_end_io(struct btrfs_raid_bio *rbio)
} else {
page = rbio_stripe_page(rbio, stripe, pagenr);
}
- kunmap(page);
+ kunmap_thread(page);
}
}
@@ -2379,18 +2379,18 @@ static noinline void finish_parity_scrub(struct btrfs_raid_bio *rbio,
/* first collect one page from each data stripe */
for (stripe = 0; stripe < nr_data; stripe++) {
p = page_in_rbio(rbio, stripe, pagenr, 0);
- pointers[stripe] = kmap(p);
+ pointers[stripe] = kmap_thread(p);
}
/* then add the parity stripe */
- pointers[stripe++] = kmap(p_page);
+ pointers[stripe++] = kmap_thread(p_page);
if (has_qstripe) {
/*
* raid6, add the qstripe and call the
* library function to fill in our p/q
*/
- pointers[stripe++] = kmap(q_page);
+ pointers[stripe++] = kmap_thread(q_page);
raid6_call.gen_syndrome(rbio->real_stripes, PAGE_SIZE,
pointers);
@@ -2402,17 +2402,17 @@ static noinline void finish_parity_scrub(struct btrfs_raid_bio *rbio,
/* Check scrubbing parity and repair it */
p = rbio_stripe_page(rbio, rbio->scrubp, pagenr);
- parity = kmap(p);
+ parity = kmap_thread(p);
if (memcmp(parity, pointers[rbio->scrubp], PAGE_SIZE))
copy_page(parity, pointers[rbio->scrubp]);
else
/* Parity is right, needn't writeback */
bitmap_clear(rbio->dbitmap, pagenr, 1);
- kunmap(p);
+ kunmap_thread(p);
for (stripe = 0; stripe < nr_data; stripe++)
- kunmap(page_in_rbio(rbio, stripe, pagenr, 0));
- kunmap(p_page);
+ kunmap_thread(page_in_rbio(rbio, stripe, pagenr, 0));
+ kunmap_thread(p_page);
}
__free_page(p_page);
diff --git a/fs/btrfs/reflink.c b/fs/btrfs/reflink.c
index 5cd02514cf4d..10e53d7eba8c 100644
--- a/fs/btrfs/reflink.c
+++ b/fs/btrfs/reflink.c
@@ -92,10 +92,10 @@ static int copy_inline_to_page(struct inode *inode,
if (comp_type == BTRFS_COMPRESS_NONE) {
char *map;
- map = kmap(page);
+ map = kmap_thread(page);
memcpy(map, data_start, datal);
flush_dcache_page(page);
- kunmap(page);
+ kunmap_thread(page);
} else {
ret = btrfs_decompress(comp_type, data_start, page, 0,
inline_size, datal);
@@ -119,10 +119,10 @@ static int copy_inline_to_page(struct inode *inode,
if (datal < block_size) {
char *map;
- map = kmap(page);
+ map = kmap_thread(page);
memset(map + datal, 0, block_size - datal);
flush_dcache_page(page);
- kunmap(page);
+ kunmap_thread(page);
}
SetPageUptodate(page);
diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index d9813a5b075a..06c383d3dc43 100644
--- a/fs/btrfs/send.c
+++ b/fs/btrfs/send.c
@@ -4863,9 +4863,9 @@ static ssize_t fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len)
}
}
- addr = kmap(page);
+ addr = kmap_thread(page);
memcpy(sctx->read_buf + ret, addr + pg_offset, cur_len);
- kunmap(page);
+ kunmap_thread(page);
unlock_page(page);
put_page(page);
index++;
diff --git a/fs/btrfs/zlib.c b/fs/btrfs/zlib.c
index 05615a1099db..45b7a907bab3 100644
--- a/fs/btrfs/zlib.c
+++ b/fs/btrfs/zlib.c
@@ -126,7 +126,7 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping,
ret = -ENOMEM;
goto out;
}
- cpage_out = kmap(out_page);
+ cpage_out = kmap_thread(out_page);
pages[0] = out_page;
nr_pages = 1;
@@ -149,12 +149,12 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping,
for (i = 0; i < in_buf_pages; i++) {
if (in_page) {
- kunmap(in_page);
+ kunmap_thread(in_page);
put_page(in_page);
}
in_page = find_get_page(mapping,
start >> PAGE_SHIFT);
- data_in = kmap(in_page);
+ data_in = kmap_thread(in_page);
memcpy(workspace->buf + i * PAGE_SIZE,
data_in, PAGE_SIZE);
start += PAGE_SIZE;
@@ -162,12 +162,12 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping,
workspace->strm.next_in = workspace->buf;
} else {
if (in_page) {
- kunmap(in_page);
+ kunmap_thread(in_page);
put_page(in_page);
}
in_page = find_get_page(mapping,
start >> PAGE_SHIFT);
- data_in = kmap(in_page);
+ data_in = kmap_thread(in_page);
start += PAGE_SIZE;
workspace->strm.next_in = data_in;
}
@@ -196,7 +196,7 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping,
* the stream end if required
*/
if (workspace->strm.avail_out == 0) {
- kunmap(out_page);
+ kunmap_thread(out_page);
if (nr_pages == nr_dest_pages) {
out_page = NULL;
ret = -E2BIG;
@@ -207,7 +207,7 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping,
ret = -ENOMEM;
goto out;
}
- cpage_out = kmap(out_page);
+ cpage_out = kmap_thread(out_page);
pages[nr_pages] = out_page;
nr_pages++;
workspace->strm.avail_out = PAGE_SIZE;
@@ -234,7 +234,7 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping,
goto out;
} else if (workspace->strm.avail_out == 0) {
/* get another page for the stream end */
- kunmap(out_page);
+ kunmap_thread(out_page);
if (nr_pages == nr_dest_pages) {
out_page = NULL;
ret = -E2BIG;
@@ -245,7 +245,7 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping,
ret = -ENOMEM;
goto out;
}
- cpage_out = kmap(out_page);
+ cpage_out = kmap_thread(out_page);
pages[nr_pages] = out_page;
nr_pages++;
workspace->strm.avail_out = PAGE_SIZE;
@@ -265,10 +265,10 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping,
out:
*out_pages = nr_pages;
if (out_page)
- kunmap(out_page);
+ kunmap_thread(out_page);
if (in_page) {
- kunmap(in_page);
+ kunmap_thread(in_page);
put_page(in_page);
}
return ret;
@@ -289,7 +289,7 @@ int zlib_decompress_bio(struct list_head *ws, struct compressed_bio *cb)
u64 disk_start = cb->start;
struct bio *orig_bio = cb->orig_bio;
- data_in = kmap(pages_in[page_in_index]);
+ data_in = kmap_thread(pages_in[page_in_index]);
workspace->strm.next_in = data_in;
workspace->strm.avail_in = min_t(size_t, srclen, PAGE_SIZE);
workspace->strm.total_in = 0;
@@ -311,7 +311,7 @@ int zlib_decompress_bio(struct list_head *ws, struct compressed_bio *cb)
if (Z_OK != zlib_inflateInit2(&workspace->strm, wbits)) {
pr_warn("BTRFS: inflateInit failed\n");
- kunmap(pages_in[page_in_index]);
+ kunmap_thread(pages_in[page_in_index]);
return -EIO;
}
while (workspace->strm.total_in < srclen) {
@@ -339,13 +339,13 @@ int zlib_decompress_bio(struct list_head *ws, struct compressed_bio *cb)
if (workspace->strm.avail_in == 0) {
unsigned long tmp;
- kunmap(pages_in[page_in_index]);
+ kunmap_thread(pages_in[page_in_index]);
page_in_index++;
if (page_in_index >= total_pages_in) {
data_in = NULL;
break;
}
- data_in = kmap(pages_in[page_in_index]);
+ data_in = kmap_thread(pages_in[page_in_index]);
workspace->strm.next_in = data_in;
tmp = srclen - workspace->strm.total_in;
workspace->strm.avail_in = min(tmp,
@@ -359,7 +359,7 @@ int zlib_decompress_bio(struct list_head *ws, struct compressed_bio *cb)
done:
zlib_inflateEnd(&workspace->strm);
if (data_in)
- kunmap(pages_in[page_in_index]);
+ kunmap_thread(pages_in[page_in_index]);
if (!ret)
zero_fill_bio(orig_bio);
return ret;
diff --git a/fs/btrfs/zstd.c b/fs/btrfs/zstd.c
index 9a4871636c6c..48e03f6dcef7 100644
--- a/fs/btrfs/zstd.c
+++ b/fs/btrfs/zstd.c
@@ -399,7 +399,7 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping,
/* map in the first page of input data */
in_page = find_get_page(mapping, start >> PAGE_SHIFT);
- workspace->in_buf.src = kmap(in_page);
+ workspace->in_buf.src = kmap_thread(in_page);
workspace->in_buf.pos = 0;
workspace->in_buf.size = min_t(size_t, len, PAGE_SIZE);
@@ -411,7 +411,7 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping,
goto out;
}
pages[nr_pages++] = out_page;
- workspace->out_buf.dst = kmap(out_page);
+ workspace->out_buf.dst = kmap_thread(out_page);
workspace->out_buf.pos = 0;
workspace->out_buf.size = min_t(size_t, max_out, PAGE_SIZE);
@@ -446,7 +446,7 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping,
if (workspace->out_buf.pos == workspace->out_buf.size) {
tot_out += PAGE_SIZE;
max_out -= PAGE_SIZE;
- kunmap(out_page);
+ kunmap_thread(out_page);
if (nr_pages == nr_dest_pages) {
out_page = NULL;
ret = -E2BIG;
@@ -458,7 +458,7 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping,
goto out;
}
pages[nr_pages++] = out_page;
- workspace->out_buf.dst = kmap(out_page);
+ workspace->out_buf.dst = kmap_thread(out_page);
workspace->out_buf.pos = 0;
workspace->out_buf.size = min_t(size_t, max_out,
PAGE_SIZE);
@@ -479,7 +479,7 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping,
start += PAGE_SIZE;
len -= PAGE_SIZE;
in_page = find_get_page(mapping, start >> PAGE_SHIFT);
- workspace->in_buf.src = kmap(in_page);
+ workspace->in_buf.src = kmap_thread(in_page);
workspace->in_buf.pos = 0;
workspace->in_buf.size = min_t(size_t, len, PAGE_SIZE);
}
@@ -518,7 +518,7 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping,
goto out;
}
pages[nr_pages++] = out_page;
- workspace->out_buf.dst = kmap(out_page);
+ workspace->out_buf.dst = kmap_thread(out_page);
workspace->out_buf.pos = 0;
workspace->out_buf.size = min_t(size_t, max_out, PAGE_SIZE);
}
@@ -565,7 +565,7 @@ int zstd_decompress_bio(struct list_head *ws, struct compressed_bio *cb)
goto done;
}
- workspace->in_buf.src = kmap(pages_in[page_in_index]);
+ workspace->in_buf.src = kmap_thread(pages_in[page_in_index]);
workspace->in_buf.pos = 0;
workspace->in_buf.size = min_t(size_t, srclen, PAGE_SIZE);
@@ -601,14 +601,14 @@ int zstd_decompress_bio(struct list_head *ws, struct compressed_bio *cb)
break;
if (workspace->in_buf.pos == workspace->in_buf.size) {
- kunmap(pages_in[page_in_index++]);
+ kunmap_thread(pages_in[page_in_index++]);
if (page_in_index >= total_pages_in) {
workspace->in_buf.src = NULL;
ret = -EIO;
goto done;
}
srclen -= PAGE_SIZE;
- workspace->in_buf.src = kmap(pages_in[page_in_index]);
+ workspace->in_buf.src = kmap_thread(pages_in[page_in_index]);
workspace->in_buf.pos = 0;
workspace->in_buf.size = min_t(size_t, srclen, PAGE_SIZE);
}
@@ -617,7 +617,7 @@ int zstd_decompress_bio(struct list_head *ws, struct compressed_bio *cb)
zero_fill_bio(orig_bio);
done:
if (workspace->in_buf.src)
- kunmap(pages_in[page_in_index]);
+ kunmap_thread(pages_in[page_in_index]);
return ret;
}
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 11/58] drivers/net: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, Jesse Brandeburg, amd-gfx, linux-afs, cluster-devel,
linux-cachefs, intel-wired-lan, Jakub Kicinski, linux-ext4,
Fenghua Yu, linux-um, intel-gfx, ecryptfs, linux-erofs,
reiserfs-devel, linux-block, linux-bcache, xen-devel,
Dan Williams, io-uring, linux-nfs, linux-ntfs-dev, netdev, kexec,
linux-kernel, linux-f2fs-devel, linux-fsdevel, bpf, linuxppc-dev,
David S. Miller, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in these drivers are localized to a single thread. To
avoid the over head of global PKRS updates use the new kmap_thread()
call.
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Jakub Kicinski <kuba@kernel.org>
Cc: Jesse Brandeburg <jesse.brandeburg@intel.com>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
drivers/net/ethernet/intel/igb/igb_ethtool.c | 4 ++--
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/intel/igb/igb_ethtool.c b/drivers/net/ethernet/intel/igb/igb_ethtool.c
index 6e8231c1ddf0..ac9189752012 100644
--- a/drivers/net/ethernet/intel/igb/igb_ethtool.c
+++ b/drivers/net/ethernet/intel/igb/igb_ethtool.c
@@ -1794,14 +1794,14 @@ static int igb_check_lbtest_frame(struct igb_rx_buffer *rx_buffer,
frame_size >>= 1;
- data = kmap(rx_buffer->page);
+ data = kmap_thread(rx_buffer->page);
if (data[3] != 0xFF ||
data[frame_size + 10] != 0xBE ||
data[frame_size + 12] != 0xAF)
match = false;
- kunmap(rx_buffer->page);
+ kunmap_thread(rx_buffer->page);
return match;
}
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
index 71ec908266a6..7d469425f8b4 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
@@ -1963,14 +1963,14 @@ static bool ixgbe_check_lbtest_frame(struct ixgbe_rx_buffer *rx_buffer,
frame_size >>= 1;
- data = kmap(rx_buffer->page) + rx_buffer->page_offset;
+ data = kmap_thread(rx_buffer->page) + rx_buffer->page_offset;
if (data[3] != 0xFF ||
data[frame_size + 10] != 0xBE ||
data[frame_size + 12] != 0xAF)
match = false;
- kunmap(rx_buffer->page);
+ kunmap_thread(rx_buffer->page);
return match;
}
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 12/58] fs/afs: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, David Howells, linux-mm, target-devel, linux-mtd,
linux-kselftest, samba-technical, Ira Weiny, ceph-devel, drbd-dev,
devel, linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm,
linux-rdma, x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-um,
intel-gfx, ecryptfs, linux-erofs, reiserfs-devel, linux-block,
linux-bcache, Dan Williams, io-uring, linux-nfs, linux-ntfs-dev,
netdev, kexec, linux-kernel, linux-f2fs-devel, linux-fsdevel, bpf,
linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this FS are localized to a single thread. To
avoid the over head of global PKRS updates use the new kmap_thread()
call.
Cc: David Howells <dhowells@redhat.com>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
fs/afs/dir.c | 16 ++++++++--------
fs/afs/dir_edit.c | 16 ++++++++--------
fs/afs/mntpt.c | 4 ++--
fs/afs/write.c | 4 ++--
4 files changed, 20 insertions(+), 20 deletions(-)
diff --git a/fs/afs/dir.c b/fs/afs/dir.c
index 1d2e61e0ab04..5d01cdb590de 100644
--- a/fs/afs/dir.c
+++ b/fs/afs/dir.c
@@ -127,14 +127,14 @@ static bool afs_dir_check_page(struct afs_vnode *dvnode, struct page *page,
qty /= sizeof(union afs_xdr_dir_block);
/* check them */
- dbuf = kmap(page);
+ dbuf = kmap_thread(page);
for (tmp = 0; tmp < qty; tmp++) {
if (dbuf->blocks[tmp].hdr.magic != AFS_DIR_MAGIC) {
printk("kAFS: %s(%lx): bad magic %d/%d is %04hx\n",
__func__, dvnode->vfs_inode.i_ino, tmp, qty,
ntohs(dbuf->blocks[tmp].hdr.magic));
trace_afs_dir_check_failed(dvnode, off, i_size);
- kunmap(page);
+ kunmap_thread(page);
trace_afs_file_error(dvnode, -EIO, afs_file_error_dir_bad_magic);
goto error;
}
@@ -146,7 +146,7 @@ static bool afs_dir_check_page(struct afs_vnode *dvnode, struct page *page,
((u8 *)&dbuf->blocks[tmp])[AFS_DIR_BLOCK_SIZE - 1] = 0;
}
- kunmap(page);
+ kunmap_thread(page);
checked:
afs_stat_v(dvnode, n_read_dir);
@@ -177,13 +177,13 @@ static bool afs_dir_check_pages(struct afs_vnode *dvnode, struct afs_read *req)
req->pos, req->index, req->nr_pages, req->offset);
for (i = 0; i < req->nr_pages; i++) {
- dbuf = kmap(req->pages[i]);
+ dbuf = kmap_thread(req->pages[i]);
for (j = 0; j < qty; j++) {
union afs_xdr_dir_block *block = &dbuf->blocks[j];
pr_warn("[%02x] %32phN\n", i * qty + j, block);
}
- kunmap(req->pages[i]);
+ kunmap_thread(req->pages[i]);
}
return false;
}
@@ -481,7 +481,7 @@ static int afs_dir_iterate(struct inode *dir, struct dir_context *ctx,
limit = blkoff & ~(PAGE_SIZE - 1);
- dbuf = kmap(page);
+ dbuf = kmap_thread(page);
/* deal with the individual blocks stashed on this page */
do {
@@ -489,7 +489,7 @@ static int afs_dir_iterate(struct inode *dir, struct dir_context *ctx,
sizeof(union afs_xdr_dir_block)];
ret = afs_dir_iterate_block(dvnode, ctx, dblock, blkoff);
if (ret != 1) {
- kunmap(page);
+ kunmap_thread(page);
goto out;
}
@@ -497,7 +497,7 @@ static int afs_dir_iterate(struct inode *dir, struct dir_context *ctx,
} while (ctx->pos < dir->i_size && blkoff < limit);
- kunmap(page);
+ kunmap_thread(page);
ret = 0;
}
diff --git a/fs/afs/dir_edit.c b/fs/afs/dir_edit.c
index b108528bf010..35ed6828e205 100644
--- a/fs/afs/dir_edit.c
+++ b/fs/afs/dir_edit.c
@@ -218,7 +218,7 @@ void afs_edit_dir_add(struct afs_vnode *vnode,
need_slots = round_up(12 + name->len + 1 + 4, AFS_DIR_DIRENT_SIZE);
need_slots /= AFS_DIR_DIRENT_SIZE;
- meta_page = kmap(page0);
+ meta_page = kmap_thread(page0);
meta = &meta_page->blocks[0];
if (i_size == 0)
goto new_directory;
@@ -247,7 +247,7 @@ void afs_edit_dir_add(struct afs_vnode *vnode,
set_page_private(page, 1);
SetPagePrivate(page);
}
- dir_page = kmap(page);
+ dir_page = kmap_thread(page);
}
/* Abandon the edit if we got a callback break. */
@@ -284,7 +284,7 @@ void afs_edit_dir_add(struct afs_vnode *vnode,
if (page != page0) {
unlock_page(page);
- kunmap(page);
+ kunmap_thread(page);
put_page(page);
}
}
@@ -323,7 +323,7 @@ void afs_edit_dir_add(struct afs_vnode *vnode,
afs_set_contig_bits(block, slot, need_slots);
if (page != page0) {
unlock_page(page);
- kunmap(page);
+ kunmap_thread(page);
put_page(page);
}
@@ -337,7 +337,7 @@ void afs_edit_dir_add(struct afs_vnode *vnode,
out_unmap:
unlock_page(page0);
- kunmap(page0);
+ kunmap_thread(page0);
put_page(page0);
_leave("");
return;
@@ -346,7 +346,7 @@ void afs_edit_dir_add(struct afs_vnode *vnode,
trace_afs_edit_dir(vnode, why, afs_edit_dir_create_inval, 0, 0, 0, 0, name->name);
clear_bit(AFS_VNODE_DIR_VALID, &vnode->flags);
if (page != page0) {
- kunmap(page);
+ kunmap_thread(page);
put_page(page);
}
goto out_unmap;
@@ -398,7 +398,7 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,
need_slots = round_up(12 + name->len + 1 + 4, AFS_DIR_DIRENT_SIZE);
need_slots /= AFS_DIR_DIRENT_SIZE;
- meta_page = kmap(page0);
+ meta_page = kmap_thread(page0);
meta = &meta_page->blocks[0];
/* Find a page that has sufficient slots available. Each VM page
@@ -410,7 +410,7 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,
page = find_lock_page(vnode->vfs_inode.i_mapping, index);
if (!page)
goto error;
- dir_page = kmap(page);
+ dir_page = kmap_thread(page);
} else {
page = page0;
dir_page = meta_page;
diff --git a/fs/afs/mntpt.c b/fs/afs/mntpt.c
index 79bc5f1338ed..562454e2fd5c 100644
--- a/fs/afs/mntpt.c
+++ b/fs/afs/mntpt.c
@@ -139,11 +139,11 @@ static int afs_mntpt_set_params(struct fs_context *fc, struct dentry *mntpt)
return ret;
}
- buf = kmap(page);
+ buf = kmap_thread(page);
ret = -EINVAL;
if (buf[size - 1] == '.')
ret = vfs_parse_fs_string(fc, "source", buf, size - 1);
- kunmap(page);
+ kunmap_thread(page);
put_page(page);
if (ret < 0)
return ret;
diff --git a/fs/afs/write.c b/fs/afs/write.c
index 4b2265cb1891..c56e5b4db4ae 100644
--- a/fs/afs/write.c
+++ b/fs/afs/write.c
@@ -38,9 +38,9 @@ static int afs_fill_page(struct afs_vnode *vnode, struct key *key,
if (pos >= vnode->vfs_inode.i_size) {
p = pos & ~PAGE_MASK;
ASSERTCMP(p + len, <=, PAGE_SIZE);
- data = kmap(page);
+ data = kmap_thread(page);
memset(data + p, 0, len);
- kunmap(page);
+ kunmap_thread(page);
return 0;
}
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 09/58] drivers/gpu: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, David Airlie, linux-mmc,
Dave Hansen, dri-devel, linux-mm, target-devel, linux-mtd,
amd-gfx, linux-kselftest, samba-technical, Ira Weiny, ceph-devel,
drbd-dev, devel, linux-cifs, linux-nilfs, linux-scsi,
linux-nvdimm, linux-rdma, x86, Patrik Jakobsson, linux-afs,
cluster-devel, linux-cachefs, intel-wired-lan, xen-devel,
linux-ext4, Fenghua Yu, linux-um, intel-gfx, ecryptfs,
linux-erofs, reiserfs-devel, linux-block, linux-bcache,
Dan Williams, io-uring, linux-nfs, linux-ntfs-dev, netdev, kexec,
linux-kernel, linux-f2fs-devel, Daniel Vetter, linux-fsdevel, bpf,
linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
These kmap() calls in the gpu stack are localized to a single thread.
To avoid the over head of global PKRS updates use the new kmap_thread()
call.
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Patrik Jakobsson <patrik.r.jakobsson@gmail.com>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 12 ++++++------
drivers/gpu/drm/gma500/gma_display.c | 4 ++--
drivers/gpu/drm/gma500/mmu.c | 10 +++++-----
drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 4 ++--
.../gpu/drm/i915/gem/selftests/i915_gem_context.c | 4 ++--
drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c | 8 ++++----
drivers/gpu/drm/i915/gt/intel_ggtt_fencing.c | 4 ++--
drivers/gpu/drm/i915/gt/intel_gtt.c | 4 ++--
drivers/gpu/drm/i915/gt/shmem_utils.c | 4 ++--
drivers/gpu/drm/i915/i915_gem.c | 8 ++++----
drivers/gpu/drm/i915/i915_gpu_error.c | 4 ++--
drivers/gpu/drm/i915/selftests/i915_perf.c | 4 ++--
drivers/gpu/drm/radeon/radeon_ttm.c | 4 ++--
13 files changed, 37 insertions(+), 37 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index 978bae731398..bd564bccb7a3 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -2437,11 +2437,11 @@ static ssize_t amdgpu_ttm_gtt_read(struct file *f, char __user *buf,
page = adev->gart.pages[p];
if (page) {
- ptr = kmap(page);
+ ptr = kmap_thread(page);
ptr += off;
r = copy_to_user(buf, ptr, cur_size);
- kunmap(adev->gart.pages[p]);
+ kunmap_thread(adev->gart.pages[p]);
} else
r = clear_user(buf, cur_size);
@@ -2507,9 +2507,9 @@ static ssize_t amdgpu_iomem_read(struct file *f, char __user *buf,
if (p->mapping != adev->mman.bdev.dev_mapping)
return -EPERM;
- ptr = kmap(p);
+ ptr = kmap_thread(p);
r = copy_to_user(buf, ptr + off, bytes);
- kunmap(p);
+ kunmap_thread(p);
if (r)
return -EFAULT;
@@ -2558,9 +2558,9 @@ static ssize_t amdgpu_iomem_write(struct file *f, const char __user *buf,
if (p->mapping != adev->mman.bdev.dev_mapping)
return -EPERM;
- ptr = kmap(p);
+ ptr = kmap_thread(p);
r = copy_from_user(ptr + off, buf, bytes);
- kunmap(p);
+ kunmap_thread(p);
if (r)
return -EFAULT;
diff --git a/drivers/gpu/drm/gma500/gma_display.c b/drivers/gpu/drm/gma500/gma_display.c
index 3df6d6e850f5..35f4e55c941f 100644
--- a/drivers/gpu/drm/gma500/gma_display.c
+++ b/drivers/gpu/drm/gma500/gma_display.c
@@ -400,9 +400,9 @@ int gma_crtc_cursor_set(struct drm_crtc *crtc,
/* Copy the cursor to cursor mem */
tmp_dst = dev_priv->vram_addr + cursor_gt->offset;
for (i = 0; i < cursor_pages; i++) {
- tmp_src = kmap(gt->pages[i]);
+ tmp_src = kmap_thread(gt->pages[i]);
memcpy(tmp_dst, tmp_src, PAGE_SIZE);
- kunmap(gt->pages[i]);
+ kunmap_thread(gt->pages[i]);
tmp_dst += PAGE_SIZE;
}
diff --git a/drivers/gpu/drm/gma500/mmu.c b/drivers/gpu/drm/gma500/mmu.c
index 505044c9a673..fba7a3a461fd 100644
--- a/drivers/gpu/drm/gma500/mmu.c
+++ b/drivers/gpu/drm/gma500/mmu.c
@@ -192,20 +192,20 @@ struct psb_mmu_pd *psb_mmu_alloc_pd(struct psb_mmu_driver *driver,
pd->invalid_pte = 0;
}
- v = kmap(pd->dummy_pt);
+ v = kmap_thread(pd->dummy_pt);
for (i = 0; i < (PAGE_SIZE / sizeof(uint32_t)); ++i)
v[i] = pd->invalid_pte;
- kunmap(pd->dummy_pt);
+ kunmap_thread(pd->dummy_pt);
- v = kmap(pd->p);
+ v = kmap_thread(pd->p);
for (i = 0; i < (PAGE_SIZE / sizeof(uint32_t)); ++i)
v[i] = pd->invalid_pde;
- kunmap(pd->p);
+ kunmap_thread(pd->p);
clear_page(kmap(pd->dummy_page));
- kunmap(pd->dummy_page);
+ kunmap_thread(pd->dummy_page);
pd->tables = vmalloc_user(sizeof(struct psb_mmu_pt *) * 1024);
if (!pd->tables)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
index 38113d3c0138..274424795fb7 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
@@ -566,9 +566,9 @@ i915_gem_object_create_shmem_from_data(struct drm_i915_private *dev_priv,
if (err < 0)
goto fail;
- vaddr = kmap(page);
+ vaddr = kmap_thread(page);
memcpy(vaddr, data, len);
- kunmap(page);
+ kunmap_thread(page);
err = pagecache_write_end(file, file->f_mapping,
offset, len, len,
diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_context.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_context.c
index 7ffc3c751432..b466c677d007 100644
--- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_context.c
@@ -1754,7 +1754,7 @@ static int check_scratch_page(struct i915_gem_context *ctx, u32 *out)
return -EINVAL;
}
- vaddr = kmap(page);
+ vaddr = kmap_thread(page);
if (!vaddr) {
pr_err("No (mappable) scratch page!\n");
return -EINVAL;
@@ -1765,7 +1765,7 @@ static int check_scratch_page(struct i915_gem_context *ctx, u32 *out)
pr_err("Inconsistent initial state of scratch page!\n");
err = -EINVAL;
}
- kunmap(page);
+ kunmap_thread(page);
return err;
}
diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
index 9c7402ce5bf9..447df22e2e06 100644
--- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
+++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
@@ -143,7 +143,7 @@ static int check_partial_mapping(struct drm_i915_gem_object *obj,
intel_gt_flush_ggtt_writes(&to_i915(obj->base.dev)->gt);
p = i915_gem_object_get_page(obj, offset >> PAGE_SHIFT);
- cpu = kmap(p) + offset_in_page(offset);
+ cpu = kmap_thread(p) + offset_in_page(offset);
drm_clflush_virt_range(cpu, sizeof(*cpu));
if (*cpu != (u32)page) {
pr_err("Partial view for %lu [%u] (offset=%llu, size=%u [%llu, row size %u], fence=%d, tiling=%d, stride=%d) misalignment, expected write to page (%llu + %u [0x%llx]) of 0x%x, found 0x%x\n",
@@ -161,7 +161,7 @@ static int check_partial_mapping(struct drm_i915_gem_object *obj,
}
*cpu = 0;
drm_clflush_virt_range(cpu, sizeof(*cpu));
- kunmap(p);
+ kunmap_thread(p);
out:
__i915_vma_put(vma);
@@ -236,7 +236,7 @@ static int check_partial_mappings(struct drm_i915_gem_object *obj,
intel_gt_flush_ggtt_writes(&to_i915(obj->base.dev)->gt);
p = i915_gem_object_get_page(obj, offset >> PAGE_SHIFT);
- cpu = kmap(p) + offset_in_page(offset);
+ cpu = kmap_thread(p) + offset_in_page(offset);
drm_clflush_virt_range(cpu, sizeof(*cpu));
if (*cpu != (u32)page) {
pr_err("Partial view for %lu [%u] (offset=%llu, size=%u [%llu, row size %u], fence=%d, tiling=%d, stride=%d) misalignment, expected write to page (%llu + %u [0x%llx]) of 0x%x, found 0x%x\n",
@@ -254,7 +254,7 @@ static int check_partial_mappings(struct drm_i915_gem_object *obj,
}
*cpu = 0;
drm_clflush_virt_range(cpu, sizeof(*cpu));
- kunmap(p);
+ kunmap_thread(p);
if (err)
return err;
diff --git a/drivers/gpu/drm/i915/gt/intel_ggtt_fencing.c b/drivers/gpu/drm/i915/gt/intel_ggtt_fencing.c
index 7fb36b12fe7a..38da348282f1 100644
--- a/drivers/gpu/drm/i915/gt/intel_ggtt_fencing.c
+++ b/drivers/gpu/drm/i915/gt/intel_ggtt_fencing.c
@@ -731,7 +731,7 @@ static void swizzle_page(struct page *page)
char *vaddr;
int i;
- vaddr = kmap(page);
+ vaddr = kmap_thread(page);
for (i = 0; i < PAGE_SIZE; i += 128) {
memcpy(temp, &vaddr[i], 64);
@@ -739,7 +739,7 @@ static void swizzle_page(struct page *page)
memcpy(&vaddr[i + 64], temp, 64);
}
- kunmap(page);
+ kunmap_thread(page);
}
/**
diff --git a/drivers/gpu/drm/i915/gt/intel_gtt.c b/drivers/gpu/drm/i915/gt/intel_gtt.c
index 2a72cce63fd9..4cfb24e9ed62 100644
--- a/drivers/gpu/drm/i915/gt/intel_gtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gtt.c
@@ -312,9 +312,9 @@ static void poison_scratch_page(struct page *page, unsigned long size)
do {
void *vaddr;
- vaddr = kmap(page);
+ vaddr = kmap_thread(page);
memset(vaddr, POISON_FREE, PAGE_SIZE);
- kunmap(page);
+ kunmap_thread(page);
page = pfn_to_page(page_to_pfn(page) + 1);
size -= PAGE_SIZE;
diff --git a/drivers/gpu/drm/i915/gt/shmem_utils.c b/drivers/gpu/drm/i915/gt/shmem_utils.c
index 43c7acbdc79d..a40d3130cebf 100644
--- a/drivers/gpu/drm/i915/gt/shmem_utils.c
+++ b/drivers/gpu/drm/i915/gt/shmem_utils.c
@@ -142,12 +142,12 @@ static int __shmem_rw(struct file *file, loff_t off,
if (IS_ERR(page))
return PTR_ERR(page);
- vaddr = kmap(page);
+ vaddr = kmap_thread(page);
if (write)
memcpy(vaddr + offset_in_page(off), ptr, this);
else
memcpy(ptr, vaddr + offset_in_page(off), this);
- kunmap(page);
+ kunmap_thread(page);
put_page(page);
len -= this;
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 9aa3066cb75d..cae8300fd224 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -312,14 +312,14 @@ shmem_pread(struct page *page, int offset, int len, char __user *user_data,
char *vaddr;
int ret;
- vaddr = kmap(page);
+ vaddr = kmap_thread(page);
if (needs_clflush)
drm_clflush_virt_range(vaddr + offset, len);
ret = __copy_to_user(user_data, vaddr + offset, len);
- kunmap(page);
+ kunmap_thread(page);
return ret ? -EFAULT : 0;
}
@@ -708,7 +708,7 @@ shmem_pwrite(struct page *page, int offset, int len, char __user *user_data,
char *vaddr;
int ret;
- vaddr = kmap(page);
+ vaddr = kmap_thread(page);
if (needs_clflush_before)
drm_clflush_virt_range(vaddr + offset, len);
@@ -717,7 +717,7 @@ shmem_pwrite(struct page *page, int offset, int len, char __user *user_data,
if (!ret && needs_clflush_after)
drm_clflush_virt_range(vaddr + offset, len);
- kunmap(page);
+ kunmap_thread(page);
return ret ? -EFAULT : 0;
}
diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c
index 3e6cbb0d1150..aecd469b6b6e 100644
--- a/drivers/gpu/drm/i915/i915_gpu_error.c
+++ b/drivers/gpu/drm/i915/i915_gpu_error.c
@@ -1058,9 +1058,9 @@ i915_vma_coredump_create(const struct intel_gt *gt,
drm_clflush_pages(&page, 1);
- s = kmap(page);
+ s = kmap_thread(page);
ret = compress_page(compress, s, dst, false);
- kunmap(page);
+ kunmap_thread(page);
drm_clflush_pages(&page, 1);
diff --git a/drivers/gpu/drm/i915/selftests/i915_perf.c b/drivers/gpu/drm/i915/selftests/i915_perf.c
index c2d001d9c0ec..7f7ef2d056f4 100644
--- a/drivers/gpu/drm/i915/selftests/i915_perf.c
+++ b/drivers/gpu/drm/i915/selftests/i915_perf.c
@@ -307,7 +307,7 @@ static int live_noa_gpr(void *arg)
}
/* Poison the ce->vm so we detect writes not to the GGTT gt->scratch */
- scratch = kmap(ce->vm->scratch[0].base.page);
+ scratch = kmap_thread(ce->vm->scratch[0].base.page);
memset(scratch, POISON_FREE, PAGE_SIZE);
rq = intel_context_create_request(ce);
@@ -405,7 +405,7 @@ static int live_noa_gpr(void *arg)
out_rq:
i915_request_put(rq);
out_ce:
- kunmap(ce->vm->scratch[0].base.page);
+ kunmap_thread(ce->vm->scratch[0].base.page);
intel_context_put(ce);
out:
stream_destroy(stream);
diff --git a/drivers/gpu/drm/radeon/radeon_ttm.c b/drivers/gpu/drm/radeon/radeon_ttm.c
index 004344dce140..0aba0cac51e1 100644
--- a/drivers/gpu/drm/radeon/radeon_ttm.c
+++ b/drivers/gpu/drm/radeon/radeon_ttm.c
@@ -1013,11 +1013,11 @@ static ssize_t radeon_ttm_gtt_read(struct file *f, char __user *buf,
page = rdev->gart.pages[p];
if (page) {
- ptr = kmap(page);
+ ptr = kmap_thread(page);
ptr += off;
r = copy_to_user(buf, ptr, cur_size);
- kunmap(rdev->gart.pages[p]);
+ kunmap_thread(rdev->gart.pages[p]);
} else
r = clear_user(buf, cur_size);
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 08/58] drivers/firmware_loader: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, kexec, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-um,
intel-gfx, ecryptfs, linux-erofs, reiserfs-devel, linux-block,
linux-bcache, Dan Williams, io-uring, linux-nfs, linux-ntfs-dev,
netdev, linux-mmc, linux-kernel, linux-f2fs-devel,
Luis Chamberlain, linux-fsdevel, bpf, linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this driver are localized to a single thread. To
avoid the over head of global PKRS updates use the new kmap_thread()
call.
Cc: Luis Chamberlain <mcgrof@kernel.org>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
drivers/base/firmware_loader/fallback.c | 4 ++--
drivers/base/firmware_loader/main.c | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/base/firmware_loader/fallback.c b/drivers/base/firmware_loader/fallback.c
index 283ca2de76d4..22dea9ba7a37 100644
--- a/drivers/base/firmware_loader/fallback.c
+++ b/drivers/base/firmware_loader/fallback.c
@@ -322,14 +322,14 @@ static void firmware_rw(struct fw_priv *fw_priv, char *buffer,
int page_ofs = offset & (PAGE_SIZE-1);
int page_cnt = min_t(size_t, PAGE_SIZE - page_ofs, count);
- page_data = kmap(fw_priv->pages[page_nr]);
+ page_data = kmap_thread(fw_priv->pages[page_nr]);
if (read)
memcpy(buffer, page_data + page_ofs, page_cnt);
else
memcpy(page_data + page_ofs, buffer, page_cnt);
- kunmap(fw_priv->pages[page_nr]);
+ kunmap_thread(fw_priv->pages[page_nr]);
buffer += page_cnt;
offset += page_cnt;
count -= page_cnt;
diff --git a/drivers/base/firmware_loader/main.c b/drivers/base/firmware_loader/main.c
index 63b9714a0154..cc884c9f8742 100644
--- a/drivers/base/firmware_loader/main.c
+++ b/drivers/base/firmware_loader/main.c
@@ -409,11 +409,11 @@ static int fw_decompress_xz_pages(struct device *dev, struct fw_priv *fw_priv,
/* decompress onto the new allocated page */
page = fw_priv->pages[fw_priv->nr_pages - 1];
- xz_buf.out = kmap(page);
+ xz_buf.out = kmap_thread(page);
xz_buf.out_pos = 0;
xz_buf.out_size = PAGE_SIZE;
xz_ret = xz_dec_run(xz_dec, &xz_buf);
- kunmap(page);
+ kunmap_thread(page);
fw_priv->size += xz_buf.out_pos;
/* partial decompression means either end or error */
if (xz_buf.out_pos != PAGE_SIZE)
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 07/58] drivers/drbd: Utilize new kmap_thread()
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, kexec, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-um,
intel-gfx, ecryptfs, linux-erofs, reiserfs-devel, linux-block,
linux-bcache, Dan Williams, io-uring, Jens Axboe, linux-nfs,
linux-ntfs-dev, netdev, linux-mmc, linux-kernel, linux-f2fs-devel,
linux-fsdevel, bpf, linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
The kmap() calls in this driver are localized to a single thread. To
avoid the over head of global PKRS updates use the new kmap_thread()
call.
Cc: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
drivers/block/drbd/drbd_main.c | 4 ++--
drivers/block/drbd/drbd_receiver.c | 12 ++++++------
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c
index 573dbf6f0c31..f0d0c6b0745e 100644
--- a/drivers/block/drbd/drbd_main.c
+++ b/drivers/block/drbd/drbd_main.c
@@ -1532,9 +1532,9 @@ static int _drbd_no_send_page(struct drbd_peer_device *peer_device, struct page
int err;
socket = peer_device->connection->data.socket;
- addr = kmap(page) + offset;
+ addr = kmap_thread(page) + offset;
err = drbd_send_all(peer_device->connection, socket, addr, size, msg_flags);
- kunmap(page);
+ kunmap_thread(page);
if (!err)
peer_device->device->send_cnt += size >> 9;
return err;
diff --git a/drivers/block/drbd/drbd_receiver.c b/drivers/block/drbd/drbd_receiver.c
index 422363daa618..4704bc0564e2 100644
--- a/drivers/block/drbd/drbd_receiver.c
+++ b/drivers/block/drbd/drbd_receiver.c
@@ -1951,13 +1951,13 @@ read_in_block(struct drbd_peer_device *peer_device, u64 id, sector_t sector,
page = peer_req->pages;
page_chain_for_each(page) {
unsigned len = min_t(int, ds, PAGE_SIZE);
- data = kmap(page);
+ data = kmap_thread(page);
err = drbd_recv_all_warn(peer_device->connection, data, len);
if (drbd_insert_fault(device, DRBD_FAULT_RECEIVE)) {
drbd_err(device, "Fault injection: Corrupting data on receive\n");
data[0] = data[0] ^ (unsigned long)-1;
}
- kunmap(page);
+ kunmap_thread(page);
if (err) {
drbd_free_peer_req(device, peer_req);
return NULL;
@@ -1992,7 +1992,7 @@ static int drbd_drain_block(struct drbd_peer_device *peer_device, int data_size)
page = drbd_alloc_pages(peer_device, 1, 1);
- data = kmap(page);
+ data = kmap_thread(page);
while (data_size) {
unsigned int len = min_t(int, data_size, PAGE_SIZE);
@@ -2001,7 +2001,7 @@ static int drbd_drain_block(struct drbd_peer_device *peer_device, int data_size)
break;
data_size -= len;
}
- kunmap(page);
+ kunmap_thread(page);
drbd_free_pages(peer_device->device, page, 0);
return err;
}
@@ -2033,10 +2033,10 @@ static int recv_dless_read(struct drbd_peer_device *peer_device, struct drbd_req
D_ASSERT(peer_device->device, sector == bio->bi_iter.bi_sector);
bio_for_each_segment(bvec, bio, iter) {
- void *mapped = kmap(bvec.bv_page) + bvec.bv_offset;
+ void *mapped = kmap_thread(bvec.bv_page) + bvec.bv_offset;
expect = min_t(int, data_size, bvec.bv_len);
err = drbd_recv_all_warn(peer_device->connection, mapped, expect);
- kunmap(bvec.bv_page);
+ kunmap_thread(bvec.bv_page);
if (err)
return err;
data_size -= expect;
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 05/58] kmap: Introduce k[un]map_thread
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-um,
intel-gfx, ecryptfs, linux-erofs, reiserfs-devel, linux-block,
linux-bcache, Dan Williams, io-uring, linux-nfs, linux-ntfs-dev,
netdev, Randy Dunlap, kexec, linux-kernel, linux-f2fs-devel,
linux-fsdevel, bpf, linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
To correctly support the semantics of kmap() with Kernel protection keys
(PKS), kmap() may be required to set the protections on multiple
processors (globally). Enabling PKS globally can be very expensive
depending on the requested operation. Furthermore, enabling a domain
globally reduces the protection afforded by PKS.
Most kmap() (Aprox 209 of 229) callers use the map within a single thread and
have no need for the protection domain to be enabled globally. However, the
remaining callers do not follow this pattern and, as best I can tell, expect
the mapping to be 'global' and available to any thread who may access the
mapping.[1]
We don't anticipate global mappings to pmem, however in general there is a
danger in changing the semantics of kmap(). Effectively, this would cause an
unresolved page fault with little to no information about why the failure
occurred.
To resolve this a number of options were considered.
1) Attempt to change all the thread local kmap() calls to kmap_atomic()[2]
2) Introduce a flags parameter to kmap() to indicate if the mapping should be
global or not
3) Change ~20 call sites to 'kmap_global()' to indicate that they require a
global enablement of the pages.
4) Change ~209 call sites to 'kmap_thread()' to indicate that the mapping is to
be used within that thread of execution only
Option 1 is simply not feasible. Option 2 would require all of the call sites
of kmap() to change. Option 3 seems like a good minimal change but there is a
danger that new code may miss the semantic change of kmap() and not get the
behavior the developer intended. Therefore, #4 was chosen.
Subsequent patches will convert most ~90% of the kmap callers to this new call
leaving about 10% of the existing kmap callers to enable PKS globally.
Cc: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
include/linux/highmem.h | 34 ++++++++++++++++++++++++++--------
1 file changed, 26 insertions(+), 8 deletions(-)
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index 2a9806e3b8d2..ef7813544719 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -60,7 +60,7 @@ static inline void kmap_flush_tlb(unsigned long addr) { }
#endif
void *kmap_high(struct page *page);
-static inline void *kmap(struct page *page)
+static inline void *__kmap(struct page *page, bool global)
{
void *addr;
@@ -74,20 +74,20 @@ static inline void *kmap(struct page *page)
* Even non-highmem pages may have additional access protections which
* need to be checked and potentially enabled.
*/
- dev_page_enable_access(page, true);
+ dev_page_enable_access(page, global);
return addr;
}
void kunmap_high(struct page *page);
-static inline void kunmap(struct page *page)
+static inline void __kunmap(struct page *page, bool global)
{
might_sleep();
/*
* Even non-highmem pages may have additional access protections which
* need to be checked and potentially disabled.
*/
- dev_page_disable_access(page, true);
+ dev_page_disable_access(page, global);
if (!PageHighMem(page))
return;
kunmap_high(page);
@@ -160,10 +160,10 @@ static inline struct page *kmap_to_page(void *addr)
static inline unsigned long totalhigh_pages(void) { return 0UL; }
-static inline void *kmap(struct page *page)
+static inline void *__kmap(struct page *page, bool global)
{
might_sleep();
- dev_page_enable_access(page, true);
+ dev_page_enable_access(page, global);
return page_address(page);
}
@@ -171,9 +171,9 @@ static inline void kunmap_high(struct page *page)
{
}
-static inline void kunmap(struct page *page)
+static inline void __kunmap(struct page *page, bool global)
{
- dev_page_disable_access(page, true);
+ dev_page_disable_access(page, global);
#ifdef ARCH_HAS_FLUSH_ON_KUNMAP
kunmap_flush_on_unmap(page_address(page));
#endif
@@ -238,6 +238,24 @@ static inline void kmap_atomic_idx_pop(void)
#endif
+static inline void *kmap(struct page *page)
+{
+ return __kmap(page, true);
+}
+static inline void kunmap(struct page *page)
+{
+ __kunmap(page, true);
+}
+
+static inline void *kmap_thread(struct page *page)
+{
+ return __kmap(page, false);
+}
+static inline void kunmap_thread(struct page *page)
+{
+ __kunmap(page, false);
+}
+
/*
* Prevent people trying to call kunmap_atomic() as if it were kunmap()
* kunmap_atomic() should get the return value of kmap_atomic, not the page.
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 04/58] kmap: Add stray access protection for device pages
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-um,
intel-gfx, ecryptfs, linux-erofs, reiserfs-devel, linux-block,
linux-bcache, Dan Williams, io-uring, linux-nfs, linux-ntfs-dev,
netdev, Randy Dunlap, kexec, linux-kernel, linux-f2fs-devel,
linux-fsdevel, bpf, linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
Device managed pages may have additional protections. These protections
need to be removed prior to valid use by kernel users.
Check for special treatment of device managed pages in kmap and take
action if needed. We use kmap as an interface for generic kernel code
because under normal circumstances it would be a bug for general kernel
code to not use kmap prior to accessing kernel memory. Therefore, this
should allow any valid kernel users to seamlessly use these pages
without issues.
Because of the critical nature of kmap it must be pointed out that the
over head on regular DRAM is carefully implemented to be as fast as
possible. Furthermore the underlying MSR write required on device pages
when protected is better than a normal MSR write.
Specifically, WRMSR(MSR_IA32_PKRS) is not serializing but still
maintains ordering properties similar to WRPKRU. The current SDM
section on PKRS needs updating but should be the same as that of WRPKRU.
So to quote from the WRPKRU text:
WRPKRU will never execute speculatively. Memory accesses
affected by PKRU register will not execute (even speculatively)
until all prior executions of WRPKRU have completed execution
and updated the PKRU register.
Still this will make accessing pmem more expensive from the kernel but
the overhead is minimized and many pmem users access this memory through
user page mappings which are not affected at all.
Cc: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
include/linux/highmem.h | 32 +++++++++++++++++++++++++++++++-
1 file changed, 31 insertions(+), 1 deletion(-)
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index 14e6202ce47f..2a9806e3b8d2 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -8,6 +8,7 @@
#include <linux/mm.h>
#include <linux/uaccess.h>
#include <linux/hardirq.h>
+#include <linux/memremap.h>
#include <asm/cacheflush.h>
@@ -31,6 +32,20 @@ static inline void invalidate_kernel_vmap_range(void *vaddr, int size)
#include <asm/kmap_types.h>
+static inline void dev_page_enable_access(struct page *page, bool global)
+{
+ if (!page_is_access_protected(page))
+ return;
+ dev_access_enable(global);
+}
+
+static inline void dev_page_disable_access(struct page *page, bool global)
+{
+ if (!page_is_access_protected(page))
+ return;
+ dev_access_disable(global);
+}
+
#ifdef CONFIG_HIGHMEM
extern void *kmap_atomic_high_prot(struct page *page, pgprot_t prot);
extern void kunmap_atomic_high(void *kvaddr);
@@ -55,6 +70,11 @@ static inline void *kmap(struct page *page)
else
addr = kmap_high(page);
kmap_flush_tlb((unsigned long)addr);
+ /*
+ * Even non-highmem pages may have additional access protections which
+ * need to be checked and potentially enabled.
+ */
+ dev_page_enable_access(page, true);
return addr;
}
@@ -63,6 +83,11 @@ void kunmap_high(struct page *page);
static inline void kunmap(struct page *page)
{
might_sleep();
+ /*
+ * Even non-highmem pages may have additional access protections which
+ * need to be checked and potentially disabled.
+ */
+ dev_page_disable_access(page, true);
if (!PageHighMem(page))
return;
kunmap_high(page);
@@ -85,6 +110,7 @@ static inline void *kmap_atomic_prot(struct page *page, pgprot_t prot)
{
preempt_disable();
pagefault_disable();
+ dev_page_enable_access(page, false);
if (!PageHighMem(page))
return page_address(page);
return kmap_atomic_high_prot(page, prot);
@@ -137,6 +163,7 @@ static inline unsigned long totalhigh_pages(void) { return 0UL; }
static inline void *kmap(struct page *page)
{
might_sleep();
+ dev_page_enable_access(page, true);
return page_address(page);
}
@@ -146,6 +173,7 @@ static inline void kunmap_high(struct page *page)
static inline void kunmap(struct page *page)
{
+ dev_page_disable_access(page, true);
#ifdef ARCH_HAS_FLUSH_ON_KUNMAP
kunmap_flush_on_unmap(page_address(page));
#endif
@@ -155,6 +183,7 @@ static inline void *kmap_atomic(struct page *page)
{
preempt_disable();
pagefault_disable();
+ dev_page_enable_access(page, false);
return page_address(page);
}
#define kmap_atomic_prot(page, prot) kmap_atomic(page)
@@ -216,7 +245,8 @@ static inline void kmap_atomic_idx_pop(void)
#define kunmap_atomic(addr) \
do { \
BUILD_BUG_ON(__same_type((addr), struct page *)); \
- kunmap_atomic_high(addr); \
+ dev_page_disable_access(kmap_to_page(addr), false); \
+ kunmap_atomic_high(addr); \
pagefault_enable(); \
preempt_enable(); \
} while (0)
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 02/58] x86/pks/test: Add testing for global option
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-um,
intel-gfx, ecryptfs, linux-erofs, reiserfs-devel, linux-block,
linux-bcache, Dan Williams, io-uring, linux-nfs, linux-ntfs-dev,
netdev, kexec, linux-kernel, linux-f2fs-devel, linux-fsdevel, bpf,
linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
Now that PKS can be enabled globaly (for all threads) add a test which
spawns a thread and tests the same PKS functionality.
The test enables/disables PKS in 1 thread while attempting to access the
page in another thread. We use the same test array as in the 'local'
PKS testing.
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
arch/x86/mm/fault.c | 4 ++
lib/pks/pks_test.c | 128 +++++++++++++++++++++++++++++++++++++++++---
2 files changed, 124 insertions(+), 8 deletions(-)
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index 4b4ff9efa298..4c74f52fbc23 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1108,6 +1108,10 @@ static int spurious_kernel_fault_check(unsigned long error_code, pte_t *pte,
if (global_pkey_is_enabled(pte, is_write, irq_state))
return 1;
+ /*
+ * NOTE: This must be after the global_pkey_is_enabled() call
+ * to allow the fixup code to be tested.
+ */
if (handle_pks_testing(error_code, irq_state))
return 1;
diff --git a/lib/pks/pks_test.c b/lib/pks/pks_test.c
index 286c8b8457da..dfddccbe4cb6 100644
--- a/lib/pks/pks_test.c
+++ b/lib/pks/pks_test.c
@@ -154,7 +154,8 @@ static void check_exception(irqentry_state_t *irq_state)
}
/* Check the exception state */
- if (!check_pkrs(test_armed_key, PKEY_DISABLE_ACCESS)) {
+ if (!check_pkrs(test_armed_key,
+ PKEY_DISABLE_ACCESS | PKEY_DISABLE_WRITE)) {
pr_err(" FAIL: PKRS cache and MSR\n");
test_exception_ctx->pass = false;
}
@@ -308,24 +309,29 @@ static int test_it(struct pks_test_ctx *ctx, struct pks_access_test *test, void
return ret;
}
-static int run_access_test(struct pks_test_ctx *ctx,
- struct pks_access_test *test,
- void *ptr)
+static void set_protection(int pkey, enum pks_access_mode mode, bool global)
{
- switch (test->mode) {
+ switch (mode) {
case PKS_TEST_NO_ACCESS:
- pks_mknoaccess(ctx->pkey, false);
+ pks_mknoaccess(pkey, global);
break;
case PKS_TEST_RDWR:
- pks_mkrdwr(ctx->pkey, false);
+ pks_mkrdwr(pkey, global);
break;
case PKS_TEST_RDONLY:
- pks_mkread(ctx->pkey, false);
+ pks_mkread(pkey, global);
break;
default:
pr_err("BUG in test invalid mode\n");
break;
}
+}
+
+static int run_access_test(struct pks_test_ctx *ctx,
+ struct pks_access_test *test,
+ void *ptr)
+{
+ set_protection(ctx->pkey, test->mode, false);
return test_it(ctx, test, ptr);
}
@@ -516,6 +522,110 @@ static void run_exception_test(void)
pass ? "PASS" : "FAIL");
}
+struct shared_data {
+ struct mutex lock;
+ struct pks_test_ctx *ctx;
+ void *kmap_addr;
+ struct pks_access_test *test;
+};
+
+static int thread_main(void *d)
+{
+ struct shared_data *data = d;
+ struct pks_test_ctx *ctx = data->ctx;
+
+ while (!kthread_should_stop()) {
+ mutex_lock(&data->lock);
+ /*
+ * wait for the main thread to hand us the page
+ * We should be spinning so hopefully we will not have gotten
+ * the global value from a schedule in.
+ */
+ if (data->kmap_addr) {
+ if (test_it(ctx, data->test, data->kmap_addr))
+ ctx->pass = false;
+ data->kmap_addr = NULL;
+ }
+ mutex_unlock(&data->lock);
+ }
+
+ return 0;
+}
+
+static void run_thread_access_test(struct shared_data *data,
+ struct pks_test_ctx *ctx,
+ struct pks_access_test *test,
+ void *ptr)
+{
+ set_protection(ctx->pkey, test->mode, true);
+
+ pr_info("checking... mode %s; write %s\n",
+ get_mode_str(test->mode), test->write ? "TRUE" : "FALSE");
+
+ mutex_lock(&data->lock);
+ data->test = test;
+ data->kmap_addr = ptr;
+ mutex_unlock(&data->lock);
+
+ while (data->kmap_addr) {
+ msleep(10);
+ }
+}
+
+static void run_global_test(void)
+{
+ struct task_struct *other_task;
+ struct pks_test_ctx *ctx;
+ struct shared_data data;
+ bool pass = true;
+ void *ptr;
+ int i;
+
+ pr_info(" ***** BEGIN: global pkey checking\n");
+
+ /* Set up context, data pgae, and thread */
+ ctx = alloc_ctx("global pkey test");
+ if (IS_ERR(ctx)) {
+ pr_err(" FAIL: no context\n");
+ pass = false;
+ goto result;
+ }
+ ptr = alloc_test_page(ctx->pkey);
+ if (!ptr) {
+ pr_err(" FAIL: no vmalloc page\n");
+ pass = false;
+ goto free_context;
+ }
+ other_task = kthread_run(thread_main, &data, "PKRS global test");
+ if (IS_ERR(other_task)) {
+ pr_err(" FAIL: Failed to start thread\n");
+ pass = false;
+ goto free_page;
+ }
+
+ memset(&data, 0, sizeof(data));
+ mutex_init(&data.lock);
+ data.ctx = ctx;
+
+ /* Start testing */
+ ctx->pass = true;
+
+ for (i = 0; i < ARRAY_SIZE(pkey_test_ary); i++) {
+ run_thread_access_test(&data, ctx, &pkey_test_ary[i], ptr);
+ }
+
+ kthread_stop(other_task);
+ pass = ctx->pass;
+
+free_page:
+ vfree(ptr);
+free_context:
+ free_ctx(ctx);
+result:
+ pr_info(" ***** END: global pkey checking : %s\n",
+ pass ? "PASS" : "FAIL");
+}
+
static void run_all(void)
{
struct pks_test_ctx *ctx[PKS_NUM_KEYS];
@@ -538,6 +648,8 @@ static void run_all(void)
}
run_exception_test();
+
+ run_global_test();
}
static void crash_it(void)
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 01/58] x86/pks: Add a global pkrs option
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-um,
intel-gfx, ecryptfs, linux-erofs, reiserfs-devel, linux-block,
linux-bcache, Dan Williams, io-uring, linux-nfs, linux-ntfs-dev,
netdev, kexec, linux-kernel, linux-f2fs-devel, linux-fsdevel, bpf,
linuxppc-dev, linux-btrfs
In-Reply-To: <20201009195033.3208459-1-ira.weiny@intel.com>
From: Ira Weiny <ira.weiny@intel.com>
Some users, such as kmap(), sometimes requires PKS to be global.
However, updating all CPUs, and worse yet all threads is expensive.
Introduce a global PKRS state which is checked at critical times to
allow the state to enable access when global PKS is required. To
accomplish this with minimal locking; the code is carefully designed
with the following key concepts.
1) Borrow the idea of lazy TLB invalidations from the fault handler
code. When enabling PKS access we anticipate that other threads are
not yet running. However, if they are we catch the fault and clean
up the MSR value.
2) When disabling PKS access we force all MSR values across all CPU's.
This is required to block access as soon as possible.[1] However, it
is key that we never attempt to update the per-task PKS values
directly. See next point.
3) Per-task PKS values never get updated with global PKS values. This
is key to prevent locking requirements and a nearly intractable
problem of trying to update every task in the system. Here are a few
key points.
3a) The MSR value can be updated with the global PKS value if that
global value happened to change while the task was running.
3b) If the task was sleeping while the global PKS was updated then
the global value is added in when task's are scheduled.
3c) If the global PKS value restricts access the MSR is updated as
soon as possible[1] and the thread value is not updated which ensures
the thread does not retain the elevated privileges after a context
switch.
4) Follow on patches must be careful to preserve the separation of the
thread PKRS value and the MSR value.
5) Access Disable on any individual pkey is turned into (Access Disable
| Write Disable) to facilitate faster integration of the global value
into the thread local MSR through a simple '&' operation. Doing
otherwise would result in complicated individual bit manipulation for
each pkey.
[1] There is a race condition which is ignored which is required for
performance issues. This potentially allows access to a thread until
the end of it's time slice. After the context switch the global value
will be restored.
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
Documentation/core-api/protection-keys.rst | 11 +-
arch/x86/entry/common.c | 7 +
arch/x86/include/asm/pkeys.h | 6 +-
arch/x86/include/asm/pkeys_common.h | 8 +-
arch/x86/kernel/process.c | 74 +++++++-
arch/x86/mm/fault.c | 189 ++++++++++++++++-----
arch/x86/mm/pkeys.c | 88 ++++++++--
include/linux/pkeys.h | 6 +-
lib/pks/pks_test.c | 16 +-
9 files changed, 329 insertions(+), 76 deletions(-)
diff --git a/Documentation/core-api/protection-keys.rst b/Documentation/core-api/protection-keys.rst
index c60366921d60..9e8a98653e13 100644
--- a/Documentation/core-api/protection-keys.rst
+++ b/Documentation/core-api/protection-keys.rst
@@ -121,9 +121,9 @@ mapping adds that mapping to the protection domain.
int pks_key_alloc(const char * const pkey_user);
#define PAGE_KERNEL_PKEY(pkey)
#define _PAGE_KEY(pkey)
- void pks_mknoaccess(int pkey);
- void pks_mkread(int pkey);
- void pks_mkrdwr(int pkey);
+ void pks_mknoaccess(int pkey, bool global);
+ void pks_mkread(int pkey, bool global);
+ void pks_mkrdwr(int pkey, bool global);
void pks_key_free(int pkey);
pks_key_alloc() allocates keys dynamically to allow better use of the limited
@@ -141,7 +141,10 @@ _PAGE_KEY().
The pks_mk*() family of calls allows kernel users the ability to change the
protections for the domain identified by the pkey specified. 3 states are
available pks_mknoaccess(), pks_mkread(), and pks_mkrdwr() which set the access
-to none, read, and read/write respectively.
+to none, read, and read/write respectively. 'global' specifies that the
+protection should be set across all threads (logical CPU's) not just the
+current running thread/CPU. This increases the overhead of PKS and lessens the
+protection so it should be used sparingly.
Finally, pks_key_free() allows a user to return the key to the allocator for
use by others.
diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c
index 324a8fd5ac10..86ad32e0095e 100644
--- a/arch/x86/entry/common.c
+++ b/arch/x86/entry/common.c
@@ -261,12 +261,19 @@ noinstr void idtentry_exit_nmi(struct pt_regs *regs, irqentry_state_t *irq_state
* current running value and set the default PKRS value for the duration of the
* exception. Thus preventing exception handlers from having the elevated
* access of the interrupted task.
+ *
+ * NOTE That the thread saved PKRS must be preserved separately to ensure
+ * global overrides do not 'stick' on a thread.
*/
noinstr void irq_save_pkrs(irqentry_state_t *state)
{
if (!cpu_feature_enabled(X86_FEATURE_PKS))
return;
+ /*
+ * The thread_pkrs must be maintained separately to prevent global
+ * overrides from 'sticking' on a thread.
+ */
state->thread_pkrs = current->thread.saved_pkrs;
state->pkrs = this_cpu_read(pkrs_cache);
write_pkrs(INIT_PKRS_VALUE);
diff --git a/arch/x86/include/asm/pkeys.h b/arch/x86/include/asm/pkeys.h
index 79952216474e..cae0153a5480 100644
--- a/arch/x86/include/asm/pkeys.h
+++ b/arch/x86/include/asm/pkeys.h
@@ -143,9 +143,9 @@ u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags);
int pks_key_alloc(const char *const pkey_user);
void pks_key_free(int pkey);
-void pks_mknoaccess(int pkey);
-void pks_mkread(int pkey);
-void pks_mkrdwr(int pkey);
+void pks_mknoaccess(int pkey, bool global);
+void pks_mkread(int pkey, bool global);
+void pks_mkrdwr(int pkey, bool global);
#endif /* CONFIG_ARCH_HAS_SUPERVISOR_PKEYS */
diff --git a/arch/x86/include/asm/pkeys_common.h b/arch/x86/include/asm/pkeys_common.h
index 8961e2ddd6ff..e380679ba1bb 100644
--- a/arch/x86/include/asm/pkeys_common.h
+++ b/arch/x86/include/asm/pkeys_common.h
@@ -6,7 +6,12 @@
#define PKR_WD_BIT 0x2
#define PKR_BITS_PER_PKEY 2
-#define PKR_AD_KEY(pkey) (PKR_AD_BIT << ((pkey) * PKR_BITS_PER_PKEY))
+/*
+ * We must define 11b as the default to make global overrides efficient.
+ * See arch/x86/kernel/process.c where the global pkrs is factored in during
+ * context switch.
+ */
+#define PKR_AD_KEY(pkey) ((PKR_WD_BIT | PKR_AD_BIT) << ((pkey) * PKR_BITS_PER_PKEY))
/*
* Define a default PKRS value for each task.
@@ -27,6 +32,7 @@
#define PKS_NUM_KEYS 16
#ifdef CONFIG_ARCH_HAS_SUPERVISOR_PKEYS
+extern u32 pkrs_global_cache;
DECLARE_PER_CPU(u32, pkrs_cache);
noinstr void write_pkrs(u32 new_pkrs);
#else
diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
index eb3a95a69392..58edd162d9cb 100644
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -43,7 +43,7 @@
#include <asm/io_bitmap.h>
#include <asm/proto.h>
#include <asm/frame.h>
-#include <asm/pkeys_common.h>
+#include <linux/pkeys.h>
#include "process.h"
@@ -189,15 +189,83 @@ int copy_thread(unsigned long clone_flags, unsigned long sp, unsigned long arg,
}
#ifdef CONFIG_ARCH_HAS_SUPERVISOR_PKEYS
-DECLARE_PER_CPU(u32, pkrs_cache);
static inline void pks_init_task(struct task_struct *tsk)
{
/* New tasks get the most restrictive PKRS value */
tsk->thread.saved_pkrs = INIT_PKRS_VALUE;
}
+
+extern u32 pkrs_global_cache;
+
+/**
+ * The global PKRS value can only increase access. Because 01b and 11b both
+ * disable access. The following truth table is our desired result for each of
+ * the pkeys when we add in the global permissions.
+ *
+ * 00 R/W - Write enabled (all access)
+ * 10 Read - write disabled (Read only)
+ * 01 NO Acc - access disabled
+ * 11 NO Acc - also access disabled
+ *
+ * local global desired required
+ * result operation
+ * 00 00 00 &
+ * 00 10 00 &
+ * 00 01 00 &
+ * 00 11 00 &
+ *
+ * 10 00 00 &
+ * 10 10 10 &
+ * 10 01 10 ^ special case
+ * 10 11 10 &
+ *
+ * 01 00 00 &
+ * 01 10 10 ^ special case
+ * 01 01 01 &
+ * 01 11 01 &
+ *
+ * 11 00 00 &
+ * 11 10 10 &
+ * 11 01 01 &
+ * 11 11 11 &
+ *
+ * In order to eliminate the need to loop through each pkey and deal with the 2
+ * above special cases we force all 01b values to 11b through the API thus
+ * resulting in the simplified truth table below.
+ *
+ * 00 R/W - Write enabled (all access)
+ * 10 Read - write disabled (Read only)
+ * 01 NO Acc - access disabled
+ * (Not allowed in the API always use 11)
+ * 11 NO Acc - access disabled
+ *
+ * local global desired effective
+ * result operation
+ * 00 00 00 &
+ * 00 10 00 &
+ * 00 11 00 &
+ * 00 11 00 &
+ *
+ * 10 00 00 &
+ * 10 10 10 &
+ * 10 11 10 &
+ * 10 11 10 &
+ *
+ * 11 00 00 &
+ * 11 10 10 &
+ * 11 11 11 &
+ * 11 11 11 &
+ *
+ * 11 00 00 &
+ * 11 10 10 &
+ * 11 11 11 &
+ * 11 11 11 &
+ *
+ * Thus we can simply 'AND' in the global pkrs value.
+ */
static inline void pks_sched_in(void)
{
- write_pkrs(current->thread.saved_pkrs);
+ write_pkrs(current->thread.saved_pkrs & pkrs_global_cache);
}
#else
static inline void pks_init_task(struct task_struct *tsk) { }
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index dd5af9399131..4b4ff9efa298 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -32,6 +32,8 @@
#include <asm/pgtable_areas.h> /* VMALLOC_START, ... */
#include <asm/kvm_para.h> /* kvm_handle_async_pf */
+#include <asm-generic/mman-common.h>
+
#define CREATE_TRACE_POINTS
#include <asm/trace/exceptions.h>
@@ -995,9 +997,124 @@ mm_fault_error(struct pt_regs *regs, unsigned long error_code,
}
}
-static int spurious_kernel_fault_check(unsigned long error_code, pte_t *pte)
+#ifdef CONFIG_ARCH_HAS_SUPERVISOR_PKEYS
+/*
+ * check if we have had a 'global' pkey update. If so, handle this like a lazy
+ * TLB; fix up the local MSR and return
+ *
+ * See arch/x86/kernel/process.c for the explanation on how global is handled
+ * with a simple '&' operation.
+ *
+ * Also we don't update the current thread saved_pkrs because we don't want the
+ * global value to 'stick' with the thread. Rather we want this to be valid
+ * only for the remainder of this time slice. For subsequent time slices the
+ * global value will be factored in during schedule; see arch/x86/kernel/process.c
+ *
+ * Finally we have a trade off between performance and forcing a restriction of
+ * permissions across all CPUs on a global update.
+ *
+ * Given the following window.
+ *
+ * Global PKRS CPU #0 CPU #1
+ * cache MSR MSR
+ *
+ * | | |
+ * Global |----------\ | |
+ * Restriction | ------------> read | <= T1
+ * (on CPU #0) | | | |
+ * ------\ | | | |
+ * ------>| | | |
+ * | | | |
+ * Update CPU #1 |--------\ | | |
+ * | --------------\ | |
+ * | | --|------------>|
+ * Global remote | | | |
+ * MSR update | | | |
+ * (CPU 2-n) | | | |
+ * |-----> CPU's | v |
+ * local | (2-N) | local --\ |
+ * update | | update ------>|(Update <= T2
+ * ----------------\ | | Incorrect)
+ * | -----------\ | |
+ * | --->|(Update OK) |
+ * Context | | |
+ * Switch |----------\ | |
+ * | ------------> read |
+ * | | | |
+ * | | | |
+ * | | v |
+ * | | local --\ |
+ * | | update ------>|(Update
+ * | | | Correct)
+ *
+ * We allow for a larger window of the global pkey being open because global
+ * updates should be rare and we don't want to burden normal faults with having
+ * to read the global state.
+ */
+static bool global_pkey_is_enabled(pte_t *pte, bool is_write,
+ irqentry_state_t *irq_state)
+{
+ u8 pkey = pte_flags_pkey(pte->pte);
+ int pkey_shift = pkey * PKR_BITS_PER_PKEY;
+ u32 mask = (((1 << PKR_BITS_PER_PKEY) - 1) << pkey_shift);
+ u32 global = READ_ONCE(pkrs_global_cache);
+ u32 val;
+
+ /* Return early if global access is not valid */
+ val = (global & mask) >> pkey_shift;
+ if ((val & PKR_AD_BIT) || (is_write && (val & PKR_WD_BIT)))
+ return false;
+
+ irq_state->pkrs &= global;
+
+ return true;
+}
+
+#else /* !CONFIG_ARCH_HAS_SUPERVISOR_PKEYS */
+__always_inline bool global_pkey_is_enabled(pte_t *pte, bool is_write,
+ irqentry_state_t *irq_state)
+{
+ return false;
+}
+#endif /* CONFIG_ARCH_HAS_SUPERVISOR_PKEYS */
+
+#ifdef CONFIG_PKS_TESTING
+bool pks_test_callback(irqentry_state_t *irq_state);
+static bool handle_pks_testing(unsigned long hw_error_code, irqentry_state_t *irq_state)
+{
+ /*
+ * If we get a protection key exception it could be because we
+ * are running the PKS test. If so, pks_test_callback() will
+ * clear the protection mechanism and return true to indicate
+ * the fault was handled
+ */
+ return pks_test_callback(irq_state);
+}
+#else /* !CONFIG_PKS_TESTING */
+static bool handle_pks_testing(unsigned long hw_error_code, irqentry_state_t *irq_state)
+{
+ return false;
+}
+#endif /* CONFIG_PKS_TESTING */
+
+
+static int spurious_kernel_fault_check(unsigned long error_code, pte_t *pte,
+ irqentry_state_t *irq_state)
{
- if ((error_code & X86_PF_WRITE) && !pte_write(*pte))
+ bool is_write = (error_code & X86_PF_WRITE);
+
+ if (IS_ENABLED(CONFIG_ARCH_HAS_SUPERVISOR_PKEYS) &&
+ error_code & X86_PF_PK) {
+ if (global_pkey_is_enabled(pte, is_write, irq_state))
+ return 1;
+
+ if (handle_pks_testing(error_code, irq_state))
+ return 1;
+
+ return 0;
+ }
+
+ if (is_write && !pte_write(*pte))
return 0;
if ((error_code & X86_PF_INSTR) && !pte_exec(*pte))
@@ -1007,7 +1124,7 @@ static int spurious_kernel_fault_check(unsigned long error_code, pte_t *pte)
}
/*
- * Handle a spurious fault caused by a stale TLB entry.
+ * Handle a spurious fault caused by a stale TLB entry or a lazy PKRS update.
*
* This allows us to lazily refresh the TLB when increasing the
* permissions of a kernel page (RO -> RW or NX -> X). Doing it
@@ -1022,13 +1139,19 @@ static int spurious_kernel_fault_check(unsigned long error_code, pte_t *pte)
* There are no security implications to leaving a stale TLB when
* increasing the permissions on a page.
*
+ * Similarly, PKRS increases in permissions are done on a thread local level.
+ * But if the caller indicates the permission should be allowd globaly we can
+ * lazily update only those threads which fault and avoid a global IPI MSR
+ * update.
+ *
* Returns non-zero if a spurious fault was handled, zero otherwise.
*
* See Intel Developer's Manual Vol 3 Section 4.10.4.3, bullet 3
* (Optional Invalidation).
*/
static noinline int
-spurious_kernel_fault(unsigned long error_code, unsigned long address)
+spurious_kernel_fault(unsigned long error_code, unsigned long address,
+ irqentry_state_t *irq_state)
{
pgd_t *pgd;
p4d_t *p4d;
@@ -1038,17 +1161,19 @@ spurious_kernel_fault(unsigned long error_code, unsigned long address)
int ret;
/*
- * Only writes to RO or instruction fetches from NX may cause
- * spurious faults.
+ * Only PKey faults or writes to RO or instruction fetches from NX may
+ * cause spurious faults.
*
* These could be from user or supervisor accesses but the TLB
* is only lazily flushed after a kernel mapping protection
* change, so user accesses are not expected to cause spurious
* faults.
*/
- if (error_code != (X86_PF_WRITE | X86_PF_PROT) &&
- error_code != (X86_PF_INSTR | X86_PF_PROT))
- return 0;
+ if (!(error_code & X86_PF_PK)) {
+ if (error_code != (X86_PF_WRITE | X86_PF_PROT) &&
+ error_code != (X86_PF_INSTR | X86_PF_PROT))
+ return 0;
+ }
pgd = init_mm.pgd + pgd_index(address);
if (!pgd_present(*pgd))
@@ -1059,27 +1184,31 @@ spurious_kernel_fault(unsigned long error_code, unsigned long address)
return 0;
if (p4d_large(*p4d))
- return spurious_kernel_fault_check(error_code, (pte_t *) p4d);
+ return spurious_kernel_fault_check(error_code, (pte_t *) p4d,
+ irq_state);
pud = pud_offset(p4d, address);
if (!pud_present(*pud))
return 0;
if (pud_large(*pud))
- return spurious_kernel_fault_check(error_code, (pte_t *) pud);
+ return spurious_kernel_fault_check(error_code, (pte_t *) pud,
+ irq_state);
pmd = pmd_offset(pud, address);
if (!pmd_present(*pmd))
return 0;
if (pmd_large(*pmd))
- return spurious_kernel_fault_check(error_code, (pte_t *) pmd);
+ return spurious_kernel_fault_check(error_code, (pte_t *) pmd,
+ irq_state);
pte = pte_offset_kernel(pmd, address);
if (!pte_present(*pte))
return 0;
- ret = spurious_kernel_fault_check(error_code, pte);
+ ret = spurious_kernel_fault_check(error_code, pte,
+ irq_state);
if (!ret)
return 0;
@@ -1087,7 +1216,8 @@ spurious_kernel_fault(unsigned long error_code, unsigned long address)
* Make sure we have permissions in PMD.
* If not, then there's a bug in the page tables:
*/
- ret = spurious_kernel_fault_check(error_code, (pte_t *) pmd);
+ ret = spurious_kernel_fault_check(error_code, (pte_t *) pmd,
+ irq_state);
WARN_ONCE(!ret, "PMD has incorrect permission bits\n");
return ret;
@@ -1150,25 +1280,6 @@ static int fault_in_kernel_space(unsigned long address)
return address >= TASK_SIZE_MAX;
}
-#ifdef CONFIG_PKS_TESTING
-bool pks_test_callback(irqentry_state_t *irq_state);
-static bool handle_pks_testing(unsigned long hw_error_code, irqentry_state_t *irq_state)
-{
- /*
- * If we get a protection key exception it could be because we
- * are running the PKS test. If so, pks_test_callback() will
- * clear the protection mechanism and return true to indicate
- * the fault was handled.
- */
- return (hw_error_code & X86_PF_PK) && pks_test_callback(irq_state);
-}
-#else
-static bool handle_pks_testing(unsigned long hw_error_code, irqentry_state_t *irq_state)
-{
- return false;
-}
-#endif
-
/*
* Called for all faults where 'address' is part of the kernel address
* space. Might get called for faults that originate from *code* that
@@ -1186,9 +1297,6 @@ do_kern_addr_fault(struct pt_regs *regs, unsigned long hw_error_code,
!cpu_feature_enabled(X86_FEATURE_PKS))
WARN_ON_ONCE(hw_error_code & X86_PF_PK);
- if (handle_pks_testing(hw_error_code, irq_state))
- return;
-
#ifdef CONFIG_X86_32
/*
* We can fault-in kernel-space virtual memory on-demand. The
@@ -1220,8 +1328,11 @@ do_kern_addr_fault(struct pt_regs *regs, unsigned long hw_error_code,
}
#endif
- /* Was the fault spurious, caused by lazy TLB invalidation? */
- if (spurious_kernel_fault(hw_error_code, address))
+ /*
+ * Was the fault spurious; caused by lazy TLB invalidation or PKRS
+ * update?
+ */
+ if (spurious_kernel_fault(hw_error_code, address, irq_state))
return;
/* kprobes don't want to hook the spurious faults: */
@@ -1492,7 +1603,7 @@ DEFINE_IDTENTRY_RAW_ERRORCODE(exc_page_fault)
*
* Fingers crossed.
*
- * The async #PF handling code takes care of idtentry handling
+ * The async #PF handling code takes care of irqentry handling
* itself.
*/
if (kvm_handle_async_pf(regs, (u32)address))
diff --git a/arch/x86/mm/pkeys.c b/arch/x86/mm/pkeys.c
index 2431c68ef752..a45893069877 100644
--- a/arch/x86/mm/pkeys.c
+++ b/arch/x86/mm/pkeys.c
@@ -263,33 +263,84 @@ noinstr void write_pkrs(u32 new_pkrs)
}
EXPORT_SYMBOL_GPL(write_pkrs);
+/*
+ * NOTE: The pkrs_global_cache is _never_ stored in the per thread PKRS cache
+ * values [thread.saved_pkrs] by design
+ *
+ * This allows us to invalidate access on running threads immediately upon
+ * invalidate. Sleeping threads will not be enabled due to the algorithm
+ * during pkrs_sched_in()
+ */
+DEFINE_SPINLOCK(pkrs_global_cache_lock);
+u32 pkrs_global_cache = INIT_PKRS_VALUE;
+EXPORT_SYMBOL_GPL(pkrs_global_cache);
+
+static inline void update_global_pkrs(int pkey, unsigned long protection)
+{
+ int pkey_shift = pkey * PKR_BITS_PER_PKEY;
+ u32 mask = (((1 << PKR_BITS_PER_PKEY) - 1) << pkey_shift);
+ u32 old_val;
+
+ spin_lock(&pkrs_global_cache_lock);
+ old_val = (pkrs_global_cache & mask) >> pkey_shift;
+ pkrs_global_cache &= ~mask;
+ if (protection & PKEY_DISABLE_ACCESS)
+ pkrs_global_cache |= PKR_AD_BIT << pkey_shift;
+ if (protection & PKEY_DISABLE_WRITE)
+ pkrs_global_cache |= PKR_WD_BIT << pkey_shift;
+
+ /*
+ * If we are preventing access from the old value. Force the
+ * update on all running threads.
+ */
+ if (((old_val == 0) && protection) ||
+ ((old_val & PKR_WD_BIT) && (protection & PKEY_DISABLE_ACCESS))) {
+ int cpu;
+
+ for_each_online_cpu(cpu) {
+ u32 *ptr = per_cpu_ptr(&pkrs_cache, cpu);
+
+ *ptr = update_pkey_val(*ptr, pkey, protection);
+ wrmsrl_on_cpu(cpu, MSR_IA32_PKRS, *ptr);
+ put_cpu_ptr(ptr);
+ }
+ }
+ spin_unlock(&pkrs_global_cache_lock);
+}
+
/**
* Do not call this directly, see pks_mk*() below.
*
* @pkey: Key for the domain to change
* @protection: protection bits to be used
+ * @global: should this change be made globally or not.
*
* Protection utilizes the same protection bits specified for User pkeys
* PKEY_DISABLE_ACCESS
* PKEY_DISABLE_WRITE
*
*/
-static inline void pks_update_protection(int pkey, unsigned long protection)
+static inline void pks_update_protection(int pkey, unsigned long protection,
+ bool global)
{
- current->thread.saved_pkrs = update_pkey_val(current->thread.saved_pkrs,
- pkey, protection);
preempt_disable();
+ if (global)
+ update_global_pkrs(pkey, protection);
+
+ current->thread.saved_pkrs = update_pkey_val(current->thread.saved_pkrs, pkey,
+ protection);
write_pkrs(current->thread.saved_pkrs);
+
preempt_enable();
}
/**
* PKS access control functions
*
- * Change the access of the domain specified by the pkey. These are global
- * updates. They only affects the current running thread. It is undefined and
- * a bug for users to call this without having allocated a pkey and using it as
- * pkey here.
+ * Change the access of the domain specified by the pkey. These may be global
+ * updates depending on the value of global. It is undefined and a bug for
+ * users to call this without having allocated a pkey and using it as pkey
+ * here.
*
* pks_mknoaccess()
* Disable all access to the domain
@@ -299,23 +350,30 @@ static inline void pks_update_protection(int pkey, unsigned long protection)
* Make the domain Read/Write
*
* @pkey the pkey for which the access should change.
- *
+ * @global if true the access is enabled on all threads/logical cpus
*/
-void pks_mknoaccess(int pkey)
+void pks_mknoaccess(int pkey, bool global)
{
- pks_update_protection(pkey, PKEY_DISABLE_ACCESS);
+ /*
+ * We force disable access to be 11b
+ * (PKEY_DISABLE_ACCESS | PKEY_DISABLE_WRITE)
+ * instaed of 01b See arch/x86/kernel/process.c where the global pkrs
+ * is factored in during context switch.
+ */
+ pks_update_protection(pkey, PKEY_DISABLE_ACCESS | PKEY_DISABLE_WRITE,
+ global);
}
EXPORT_SYMBOL_GPL(pks_mknoaccess);
-void pks_mkread(int pkey)
+void pks_mkread(int pkey, bool global)
{
- pks_update_protection(pkey, PKEY_DISABLE_WRITE);
+ pks_update_protection(pkey, PKEY_DISABLE_WRITE, global);
}
EXPORT_SYMBOL_GPL(pks_mkread);
-void pks_mkrdwr(int pkey)
+void pks_mkrdwr(int pkey, bool global)
{
- pks_update_protection(pkey, 0);
+ pks_update_protection(pkey, 0, global);
}
EXPORT_SYMBOL_GPL(pks_mkrdwr);
@@ -377,7 +435,7 @@ void pks_key_free(int pkey)
return;
/* Restore to default of no access */
- pks_mknoaccess(pkey);
+ pks_mknoaccess(pkey, true);
pks_key_users[pkey] = NULL;
__clear_bit(pkey, &pks_key_allocation_map);
}
diff --git a/include/linux/pkeys.h b/include/linux/pkeys.h
index f9552bd9341f..8f3bfec83949 100644
--- a/include/linux/pkeys.h
+++ b/include/linux/pkeys.h
@@ -57,15 +57,15 @@ static inline int pks_key_alloc(const char * const pkey_user)
static inline void pks_key_free(int pkey)
{
}
-static inline void pks_mknoaccess(int pkey)
+static inline void pks_mknoaccess(int pkey, bool global)
{
WARN_ON_ONCE(1);
}
-static inline void pks_mkread(int pkey)
+static inline void pks_mkread(int pkey, bool global)
{
WARN_ON_ONCE(1);
}
-static inline void pks_mkrdwr(int pkey)
+static inline void pks_mkrdwr(int pkey, bool global)
{
WARN_ON_ONCE(1);
}
diff --git a/lib/pks/pks_test.c b/lib/pks/pks_test.c
index d7dbf92527bd..286c8b8457da 100644
--- a/lib/pks/pks_test.c
+++ b/lib/pks/pks_test.c
@@ -163,12 +163,12 @@ static void check_exception(irqentry_state_t *irq_state)
* Check we can update the value during exception without affecting the
* calling thread. The calling thread is checked after exception...
*/
- pks_mkrdwr(test_armed_key);
+ pks_mkrdwr(test_armed_key, false);
if (!check_pkrs(test_armed_key, 0)) {
pr_err(" FAIL: exception did not change register to 0\n");
test_exception_ctx->pass = false;
}
- pks_mknoaccess(test_armed_key);
+ pks_mknoaccess(test_armed_key, false);
if (!check_pkrs(test_armed_key, PKEY_DISABLE_ACCESS | PKEY_DISABLE_WRITE)) {
pr_err(" FAIL: exception did not change register to 0x3\n");
test_exception_ctx->pass = false;
@@ -314,13 +314,13 @@ static int run_access_test(struct pks_test_ctx *ctx,
{
switch (test->mode) {
case PKS_TEST_NO_ACCESS:
- pks_mknoaccess(ctx->pkey);
+ pks_mknoaccess(ctx->pkey, false);
break;
case PKS_TEST_RDWR:
- pks_mkrdwr(ctx->pkey);
+ pks_mkrdwr(ctx->pkey, false);
break;
case PKS_TEST_RDONLY:
- pks_mkread(ctx->pkey);
+ pks_mkread(ctx->pkey, false);
break;
default:
pr_err("BUG in test invalid mode\n");
@@ -476,7 +476,7 @@ static void run_exception_test(void)
goto free_context;
}
- pks_mkread(ctx->pkey);
+ pks_mkread(ctx->pkey, false);
spin_lock(&test_lock);
WRITE_ONCE(test_exception_ctx, ctx);
@@ -556,7 +556,7 @@ static void crash_it(void)
return;
}
- pks_mknoaccess(ctx->pkey);
+ pks_mknoaccess(ctx->pkey, false);
spin_lock(&test_lock);
WRITE_ONCE(test_armed_key, 0);
@@ -618,7 +618,7 @@ static ssize_t pks_write_file(struct file *file, const char __user *user_buf,
/* start of context switch test */
if (!strcmp(buf, "1")) {
/* Ensure a known state to test context switch */
- pks_mknoaccess(ctx->pkey);
+ pks_mknoaccess(ctx->pkey, false);
}
/* After context switch msr should be restored */
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply related
* [PATCH RFC PKS/PMEM 00/58] PMEM: Introduce stray write protection for PMEM
From: ira.weiny @ 2020-10-09 19:49 UTC (permalink / raw)
To: Andrew Morton, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Andy Lutomirski, Peter Zijlstra
Cc: linux-aio, linux-efi, kvm, linux-doc, linux-mmc, Dave Hansen,
dri-devel, linux-mm, target-devel, linux-mtd, linux-kselftest,
samba-technical, Ira Weiny, ceph-devel, drbd-dev, devel,
linux-cifs, linux-nilfs, linux-scsi, linux-nvdimm, linux-rdma,
x86, amd-gfx, linux-afs, cluster-devel, linux-cachefs,
intel-wired-lan, xen-devel, linux-ext4, Fenghua Yu, linux-um,
intel-gfx, ecryptfs, linux-erofs, reiserfs-devel, linux-block,
linux-bcache, Dan Williams, io-uring, linux-nfs, linux-ntfs-dev,
netdev, kexec, linux-kernel, linux-f2fs-devel, linux-fsdevel, bpf,
linuxppc-dev, linux-btrfs
From: Ira Weiny <ira.weiny@intel.com>
Should a stray write in the kernel occur persistent memory is affected more
than regular memory. A write to the wrong area of memory could result in
latent data corruption which will will persist after a reboot. PKS provides a
nice way to restrict access to persistent memory kernel mappings, while
providing fast access when needed.
Since the last RFC[1] this patch set has grown quite a bit. It now depends on
the core patches submitted separately.
https://lore.kernel.org/lkml/20201009194258.3207172-1-ira.weiny@intel.com/
And contained in the git tree here:
https://github.com/weiny2/linux-kernel/tree/pks-rfc-v3
However, functionally there is only 1 major change from the last RFC.
Specifically, kmap() is most often used within a single thread in a 'map/do
something/unmap' pattern. In fact this is the pattern used in ~90% of the
callers of kmap(). This pattern works very well for the pmem use case and the
testing which was done. However, there were another ~20-30 kmap users which do
not follow this pattern. Some of them seem to expect the mapping to be
'global' while others require a detailed audit to be sure.[2][3]
While we don't anticipate global mappings to pmem there is a danger in
changing the semantics of kmap(). Effectively, this would cause an unresolved
page fault with little to no information about why.
There were a number of options considered.
1) Attempt to change all the thread local kmap() calls to kmap_atomic()
2) Introduce a flags parameter to kmap() to indicate if the mapping should be
global or not
3) Change ~20-30 call sites to 'kmap_global()' to indicate that they require a
global mapping of the pages
4) Change ~209 call sites to 'kmap_thread()' to indicate that the mapping is to
be used within that thread of execution only
Option 1 is simply not feasible kmap_atomic() is not the same semantic as
kmap() within a single tread. Option 2 would require all of the call sites of
kmap() to change. Option 3 seems like a good minimal change but there is a
danger that new code may miss the semantic change of kmap() and not get the
behavior intended for future users. Therefore, option #4 was chosen.
To handle the global PKRS state in the most efficient manner possible. We
lazily override the thread specific PKRS key value only when needed because we
anticipate PKS to not be needed will not be needed most of the time. And even
when it is used 90% of the time it is a thread local call.
[1] https://lore.kernel.org/lkml/20200717072056.73134-1-ira.weiny@intel.com/
[2] The following list of callers continue calling kmap() (utilizing the global
PKRS). It would be nice if more of them could be converted to kmap_thread()
drivers/firewire/net.c: ptr = kmap(dev->broadcast_rcv_buffer.pages[u]);
drivers/gpu/drm/i915/gem/i915_gem_pages.c: return kmap(sg_page(sgt->sgl));
drivers/gpu/drm/ttm/ttm_bo_util.c: map->virtual = kmap(map->page);
drivers/infiniband/hw/qib/qib_user_sdma.c: mpage = kmap(page);
drivers/misc/vmw_vmci/vmci_host.c: context->notify = kmap(context->notify_page) + (uva & (PAGE_SIZE - 1));
drivers/misc/xilinx_sdfec.c: addr = kmap(pages[i]);
drivers/mmc/host/usdhi6rol0.c: host->pg.mapped = kmap(host->pg.page);
drivers/mmc/host/usdhi6rol0.c: host->pg.mapped = kmap(host->pg.page);
drivers/mmc/host/usdhi6rol0.c: host->pg.mapped = kmap(host->pg.page);
drivers/nvme/target/tcp.c: iov->iov_base = kmap(sg_page(sg)) + sg->offset + sg_offset;
drivers/scsi/libiscsi_tcp.c: segment->sg_mapped = kmap(sg_page(sg));
drivers/target/iscsi/iscsi_target.c: iov[i].iov_base = kmap(sg_page(sg)) + sg->offset + page_off;
drivers/target/target_core_transport.c: return kmap(sg_page(sg)) + sg->offset;
fs/btrfs/check-integrity.c: block_ctx->datav[i] = kmap(block_ctx->pagev[i]);
fs/ceph/dir.c: cache_ctl->dentries = kmap(cache_ctl->page);
fs/ceph/inode.c: ctl->dentries = kmap(ctl->page);
fs/erofs/zpvec.h: kmap_atomic(ctor->curr) : kmap(ctor->curr);
lib/scatterlist.c: miter->addr = kmap(miter->page) + miter->__offset;
net/ceph/pagelist.c: pl->mapped_tail = kmap(page);
net/ceph/pagelist.c: pl->mapped_tail = kmap(page);
virt/kvm/kvm_main.c: hva = kmap(page);
[3] The following appear to follow the same pattern as ext2 which was converted
after some code audit. So I _think_ they too could be converted to
k[un]map_thread().
fs/freevxfs/vxfs_subr.c|75| kmap(pp);
fs/jfs/jfs_metapage.c|102| kmap(page);
fs/jfs/jfs_metapage.c|156| kmap(page);
fs/minix/dir.c|72| kmap(page);
fs/nilfs2/dir.c|195| kmap(page);
fs/nilfs2/ifile.h|24| void *kaddr = kmap(ibh->b_page);
fs/ntfs/aops.h|78| kmap(page);
fs/ntfs/compress.c|574| kmap(page);
fs/qnx6/dir.c|32| kmap(page);
fs/qnx6/dir.c|58| kmap(*p = page);
fs/qnx6/inode.c|190| kmap(page);
fs/qnx6/inode.c|557| kmap(page);
fs/reiserfs/inode.c|2397| kmap(bh_result->b_page);
fs/reiserfs/xattr.c|444| kmap(page);
fs/sysv/dir.c|60| kmap(page);
fs/sysv/dir.c|262| kmap(page);
fs/ufs/dir.c|194| kmap(page);
fs/ufs/dir.c|562| kmap(page);
Ira Weiny (58):
x86/pks: Add a global pkrs option
x86/pks/test: Add testing for global option
memremap: Add zone device access protection
kmap: Add stray access protection for device pages
kmap: Introduce k[un]map_thread
kmap: Introduce k[un]map_thread debugging
drivers/drbd: Utilize new kmap_thread()
drivers/firmware_loader: Utilize new kmap_thread()
drivers/gpu: Utilize new kmap_thread()
drivers/rdma: Utilize new kmap_thread()
drivers/net: Utilize new kmap_thread()
fs/afs: Utilize new kmap_thread()
fs/btrfs: Utilize new kmap_thread()
fs/cifs: Utilize new kmap_thread()
fs/ecryptfs: Utilize new kmap_thread()
fs/gfs2: Utilize new kmap_thread()
fs/nilfs2: Utilize new kmap_thread()
fs/hfs: Utilize new kmap_thread()
fs/hfsplus: Utilize new kmap_thread()
fs/jffs2: Utilize new kmap_thread()
fs/nfs: Utilize new kmap_thread()
fs/f2fs: Utilize new kmap_thread()
fs/fuse: Utilize new kmap_thread()
fs/freevxfs: Utilize new kmap_thread()
fs/reiserfs: Utilize new kmap_thread()
fs/zonefs: Utilize new kmap_thread()
fs/ubifs: Utilize new kmap_thread()
fs/cachefiles: Utilize new kmap_thread()
fs/ntfs: Utilize new kmap_thread()
fs/romfs: Utilize new kmap_thread()
fs/vboxsf: Utilize new kmap_thread()
fs/hostfs: Utilize new kmap_thread()
fs/cramfs: Utilize new kmap_thread()
fs/erofs: Utilize new kmap_thread()
fs: Utilize new kmap_thread()
fs/ext2: Use ext2_put_page
fs/ext2: Utilize new kmap_thread()
fs/isofs: Utilize new kmap_thread()
fs/jffs2: Utilize new kmap_thread()
net: Utilize new kmap_thread()
drivers/target: Utilize new kmap_thread()
drivers/scsi: Utilize new kmap_thread()
drivers/mmc: Utilize new kmap_thread()
drivers/xen: Utilize new kmap_thread()
drivers/firmware: Utilize new kmap_thread()
drives/staging: Utilize new kmap_thread()
drivers/mtd: Utilize new kmap_thread()
drivers/md: Utilize new kmap_thread()
drivers/misc: Utilize new kmap_thread()
drivers/android: Utilize new kmap_thread()
kernel: Utilize new kmap_thread()
mm: Utilize new kmap_thread()
lib: Utilize new kmap_thread()
powerpc: Utilize new kmap_thread()
samples: Utilize new kmap_thread()
dax: Stray access protection for dax_direct_access()
nvdimm/pmem: Stray access protection for pmem->virt_addr
[dax|pmem]: Enable stray access protection
Documentation/core-api/protection-keys.rst | 11 +-
arch/powerpc/mm/mem.c | 4 +-
arch/x86/entry/common.c | 28 +++
arch/x86/include/asm/pkeys.h | 6 +-
arch/x86/include/asm/pkeys_common.h | 8 +-
arch/x86/kernel/process.c | 74 ++++++-
arch/x86/mm/fault.c | 193 ++++++++++++++----
arch/x86/mm/pkeys.c | 88 ++++++--
drivers/android/binder_alloc.c | 4 +-
drivers/base/firmware_loader/fallback.c | 4 +-
drivers/base/firmware_loader/main.c | 4 +-
drivers/block/drbd/drbd_main.c | 4 +-
drivers/block/drbd/drbd_receiver.c | 12 +-
drivers/dax/device.c | 2 +
drivers/dax/super.c | 2 +
drivers/firmware/efi/capsule-loader.c | 6 +-
drivers/firmware/efi/capsule.c | 4 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 12 +-
drivers/gpu/drm/gma500/gma_display.c | 4 +-
drivers/gpu/drm/gma500/mmu.c | 10 +-
drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 4 +-
.../drm/i915/gem/selftests/i915_gem_context.c | 4 +-
.../drm/i915/gem/selftests/i915_gem_mman.c | 8 +-
drivers/gpu/drm/i915/gt/intel_ggtt_fencing.c | 4 +-
drivers/gpu/drm/i915/gt/intel_gtt.c | 4 +-
drivers/gpu/drm/i915/gt/shmem_utils.c | 4 +-
drivers/gpu/drm/i915/i915_gem.c | 8 +-
drivers/gpu/drm/i915/i915_gpu_error.c | 4 +-
drivers/gpu/drm/i915/selftests/i915_perf.c | 4 +-
drivers/gpu/drm/radeon/radeon_ttm.c | 4 +-
drivers/infiniband/hw/hfi1/sdma.c | 4 +-
drivers/infiniband/hw/i40iw/i40iw_cm.c | 10 +-
drivers/infiniband/sw/siw/siw_qp_tx.c | 14 +-
drivers/md/bcache/request.c | 4 +-
drivers/misc/vmw_vmci/vmci_queue_pair.c | 12 +-
drivers/mmc/host/mmc_spi.c | 4 +-
drivers/mmc/host/sdricoh_cs.c | 4 +-
drivers/mtd/mtd_blkdevs.c | 12 +-
drivers/net/ethernet/intel/igb/igb_ethtool.c | 4 +-
.../net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 4 +-
drivers/nvdimm/pmem.c | 6 +
drivers/scsi/ipr.c | 8 +-
drivers/scsi/pmcraid.c | 8 +-
drivers/staging/rts5208/rtsx_transport.c | 4 +-
drivers/target/target_core_iblock.c | 4 +-
drivers/target/target_core_rd.c | 4 +-
drivers/target/target_core_transport.c | 4 +-
drivers/xen/gntalloc.c | 4 +-
fs/afs/dir.c | 16 +-
fs/afs/dir_edit.c | 16 +-
fs/afs/mntpt.c | 4 +-
fs/afs/write.c | 4 +-
fs/aio.c | 4 +-
fs/binfmt_elf.c | 4 +-
fs/binfmt_elf_fdpic.c | 4 +-
fs/btrfs/check-integrity.c | 4 +-
fs/btrfs/compression.c | 4 +-
fs/btrfs/inode.c | 16 +-
fs/btrfs/lzo.c | 24 +--
fs/btrfs/raid56.c | 34 +--
fs/btrfs/reflink.c | 8 +-
fs/btrfs/send.c | 4 +-
fs/btrfs/zlib.c | 32 +--
fs/btrfs/zstd.c | 20 +-
fs/cachefiles/rdwr.c | 4 +-
fs/cifs/cifsencrypt.c | 6 +-
fs/cifs/file.c | 16 +-
fs/cifs/smb2ops.c | 8 +-
fs/cramfs/inode.c | 10 +-
fs/ecryptfs/crypto.c | 8 +-
fs/ecryptfs/read_write.c | 8 +-
fs/erofs/super.c | 4 +-
fs/erofs/xattr.c | 4 +-
fs/exec.c | 10 +-
fs/ext2/dir.c | 8 +-
fs/ext2/ext2.h | 8 +
fs/ext2/namei.c | 15 +-
fs/f2fs/f2fs.h | 8 +-
fs/freevxfs/vxfs_immed.c | 4 +-
fs/fuse/readdir.c | 4 +-
fs/gfs2/bmap.c | 4 +-
fs/gfs2/ops_fstype.c | 4 +-
fs/hfs/bnode.c | 14 +-
fs/hfs/btree.c | 20 +-
fs/hfsplus/bitmap.c | 20 +-
fs/hfsplus/bnode.c | 102 ++++-----
fs/hfsplus/btree.c | 18 +-
fs/hostfs/hostfs_kern.c | 12 +-
fs/io_uring.c | 4 +-
fs/isofs/compress.c | 4 +-
fs/jffs2/file.c | 8 +-
fs/jffs2/gc.c | 4 +-
fs/nfs/dir.c | 20 +-
fs/nilfs2/alloc.c | 34 +--
fs/nilfs2/cpfile.c | 4 +-
fs/ntfs/aops.c | 4 +-
fs/reiserfs/journal.c | 4 +-
fs/romfs/super.c | 4 +-
fs/splice.c | 4 +-
fs/ubifs/file.c | 16 +-
fs/vboxsf/file.c | 12 +-
fs/zonefs/super.c | 4 +-
include/linux/entry-common.h | 3 +
include/linux/highmem.h | 63 +++++-
include/linux/memremap.h | 1 +
include/linux/mm.h | 43 ++++
include/linux/pkeys.h | 6 +-
include/linux/sched.h | 8 +
include/trace/events/kmap_thread.h | 56 +++++
init/init_task.c | 6 +
kernel/fork.c | 18 ++
kernel/kexec_core.c | 8 +-
lib/Kconfig.debug | 8 +
lib/iov_iter.c | 12 +-
lib/pks/pks_test.c | 138 +++++++++++--
lib/test_bpf.c | 4 +-
lib/test_hmm.c | 8 +-
mm/Kconfig | 13 ++
mm/debug.c | 23 +++
mm/memory.c | 8 +-
mm/memremap.c | 90 ++++++++
mm/swapfile.c | 4 +-
mm/userfaultfd.c | 4 +-
net/ceph/messenger.c | 4 +-
net/core/datagram.c | 4 +-
net/core/sock.c | 8 +-
net/ipv4/ip_output.c | 4 +-
net/sunrpc/cache.c | 4 +-
net/sunrpc/xdr.c | 8 +-
net/tls/tls_device.c | 4 +-
samples/vfio-mdev/mbochs.c | 4 +-
131 files changed, 1284 insertions(+), 565 deletions(-)
create mode 100644 include/trace/events/kmap_thread.h
--
2.28.0.rc0.12.gb6a658bd00c9
^ permalink raw reply
* [PATCH v4 2/2] lkdtm/powerpc: Add SLB multihit test
From: Ganesh Goudar @ 2020-10-09 6:40 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: msuchanek, Ganesh Goudar, keescook, npiggin, mahesh
In-Reply-To: <20201009064005.19777-1-ganeshgr@linux.ibm.com>
To check machine check handling, add support to inject slb
multihit errors.
Cc: Kees Cook <keescook@chromium.org>
Reviewed-by: Michal Suchánek <msuchanek@suse.de>
Co-developed-by: Mahesh Salgaonkar <mahesh@linux.ibm.com>
Signed-off-by: Mahesh Salgaonkar <mahesh@linux.ibm.com>
Signed-off-by: Ganesh Goudar <ganeshgr@linux.ibm.com>
---
drivers/misc/lkdtm/Makefile | 1 +
drivers/misc/lkdtm/core.c | 3 +
drivers/misc/lkdtm/lkdtm.h | 3 +
drivers/misc/lkdtm/powerpc.c | 156 ++++++++++++++++++++++++
tools/testing/selftests/lkdtm/tests.txt | 1 +
5 files changed, 164 insertions(+)
create mode 100644 drivers/misc/lkdtm/powerpc.c
diff --git a/drivers/misc/lkdtm/Makefile b/drivers/misc/lkdtm/Makefile
index c70b3822013f..f37ecfb0a707 100644
--- a/drivers/misc/lkdtm/Makefile
+++ b/drivers/misc/lkdtm/Makefile
@@ -10,6 +10,7 @@ lkdtm-$(CONFIG_LKDTM) += rodata_objcopy.o
lkdtm-$(CONFIG_LKDTM) += usercopy.o
lkdtm-$(CONFIG_LKDTM) += stackleak.o
lkdtm-$(CONFIG_LKDTM) += cfi.o
+lkdtm-$(CONFIG_PPC64) += powerpc.o
KASAN_SANITIZE_stackleak.o := n
KCOV_INSTRUMENT_rodata.o := n
diff --git a/drivers/misc/lkdtm/core.c b/drivers/misc/lkdtm/core.c
index a5e344df9166..8d5db42baa90 100644
--- a/drivers/misc/lkdtm/core.c
+++ b/drivers/misc/lkdtm/core.c
@@ -178,6 +178,9 @@ static const struct crashtype crashtypes[] = {
#ifdef CONFIG_X86_32
CRASHTYPE(DOUBLE_FAULT),
#endif
+#ifdef CONFIG_PPC64
+ CRASHTYPE(PPC_SLB_MULTIHIT),
+#endif
};
diff --git a/drivers/misc/lkdtm/lkdtm.h b/drivers/misc/lkdtm/lkdtm.h
index 8878538b2c13..b305bd511ee5 100644
--- a/drivers/misc/lkdtm/lkdtm.h
+++ b/drivers/misc/lkdtm/lkdtm.h
@@ -104,4 +104,7 @@ void lkdtm_STACKLEAK_ERASING(void);
/* cfi.c */
void lkdtm_CFI_FORWARD_PROTO(void);
+/* powerpc.c */
+void lkdtm_PPC_SLB_MULTIHIT(void);
+
#endif
diff --git a/drivers/misc/lkdtm/powerpc.c b/drivers/misc/lkdtm/powerpc.c
new file mode 100644
index 000000000000..f388b53dccba
--- /dev/null
+++ b/drivers/misc/lkdtm/powerpc.c
@@ -0,0 +1,156 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include "lkdtm.h"
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+
+/* Gets index for new slb entry */
+static inline unsigned long get_slb_index(void)
+{
+ unsigned long index;
+
+ index = get_paca()->stab_rr;
+
+ /*
+ * simple round-robin replacement of slb starting at SLB_NUM_BOLTED.
+ */
+ if (index < (mmu_slb_size - 1))
+ index++;
+ else
+ index = SLB_NUM_BOLTED;
+ get_paca()->stab_rr = index;
+ return index;
+}
+
+#define slb_esid_mask(ssize) \
+ (((ssize) == MMU_SEGSIZE_256M) ? ESID_MASK : ESID_MASK_1T)
+
+/* Form the operand for slbmte */
+static inline unsigned long mk_esid_data(unsigned long ea, int ssize,
+ unsigned long slot)
+{
+ return (ea & slb_esid_mask(ssize)) | SLB_ESID_V | slot;
+}
+
+#define slb_vsid_shift(ssize) \
+ ((ssize) == MMU_SEGSIZE_256M ? SLB_VSID_SHIFT : SLB_VSID_SHIFT_1T)
+
+/* Form the operand for slbmte */
+static inline unsigned long mk_vsid_data(unsigned long ea, int ssize,
+ unsigned long flags)
+{
+ return (get_kernel_vsid(ea, ssize) << slb_vsid_shift(ssize)) | flags |
+ ((unsigned long)ssize << SLB_VSID_SSIZE_SHIFT);
+}
+
+/* Inserts new slb entry */
+static void insert_slb_entry(char *p, int ssize)
+{
+ unsigned long flags, entry;
+
+ flags = SLB_VSID_KERNEL | mmu_psize_defs[MMU_PAGE_64K].sllp;
+ preempt_disable();
+
+ entry = get_slb_index();
+ asm volatile("slbmte %0,%1" :
+ : "r" (mk_vsid_data((unsigned long)p, ssize, flags)),
+ "r" (mk_esid_data((unsigned long)p, ssize, entry))
+ : "memory");
+
+ entry = get_slb_index();
+ asm volatile("slbmte %0,%1" :
+ : "r" (mk_vsid_data((unsigned long)p, ssize, flags)),
+ "r" (mk_esid_data((unsigned long)p, ssize, entry))
+ : "memory");
+ preempt_enable();
+ /*
+ * This triggers exception, If handled correctly we must recover
+ * from this error.
+ */
+ p[0] = '!';
+}
+
+/* Inject slb multihit on vmalloc-ed address i.e 0xD00... */
+static void inject_vmalloc_slb_multihit(void)
+{
+ char *p;
+
+ p = vmalloc(2048);
+ if (!p)
+ return;
+
+ insert_slb_entry(p, MMU_SEGSIZE_1T);
+ vfree(p);
+}
+
+/* Inject slb multihit on kmalloc-ed address i.e 0xC00... */
+static void inject_kmalloc_slb_multihit(void)
+{
+ char *p;
+
+ p = kmalloc(2048, GFP_KERNEL);
+ if (!p)
+ return;
+
+ insert_slb_entry(p, MMU_SEGSIZE_1T);
+ kfree(p);
+}
+
+/*
+ * Few initial SLB entries are bolted. Add a test to inject
+ * multihit in bolted entry 0.
+ */
+static void insert_dup_slb_entry_0(void)
+{
+ unsigned long test_address = 0xC000000000000000;
+ volatile unsigned long *test_ptr;
+ unsigned long entry, i = 0;
+ unsigned long esid, vsid;
+
+ test_ptr = (unsigned long *)test_address;
+ preempt_disable();
+
+ asm volatile("slbmfee %0,%1" : "=r" (esid) : "r" (i));
+ asm volatile("slbmfev %0,%1" : "=r" (vsid) : "r" (i));
+ entry = get_slb_index();
+
+ /* for i !=0 we would need to mask out the old entry number */
+ asm volatile("slbmte %0,%1" :
+ : "r" (vsid),
+ "r" (esid | entry)
+ : "memory");
+
+ asm volatile("slbmfee %0,%1" : "=r" (esid) : "r" (i));
+ asm volatile("slbmfev %0,%1" : "=r" (vsid) : "r" (i));
+ entry = get_slb_index();
+
+ /* for i !=0 we would need to mask out the old entry number */
+ asm volatile("slbmte %0,%1" :
+ : "r" (vsid),
+ "r" (esid | entry)
+ : "memory");
+
+ pr_info("%s accessing test address 0x%lx: 0x%lx\n",
+ __func__, test_address, *test_ptr);
+
+ preempt_enable();
+}
+
+void lkdtm_PPC_SLB_MULTIHIT(void)
+{
+ if (mmu_has_feature(MMU_FTR_HPTE_TABLE)) {
+ pr_info("Injecting SLB multihit errors\n");
+ /*
+ * These need not be separate tests, And they do pretty
+ * much same thing. In any case we must recover from the
+ * errors introduced by these functions, machine would not
+ * survive these tests in case of failure to handle.
+ */
+ inject_vmalloc_slb_multihit();
+ inject_kmalloc_slb_multihit();
+ insert_dup_slb_entry_0();
+ pr_info("Recovered from SLB multihit errors\n");
+ } else {
+ pr_err("XFAIL: This test is for ppc64 and with hash mode MMU only\n");
+ }
+}
diff --git a/tools/testing/selftests/lkdtm/tests.txt b/tools/testing/selftests/lkdtm/tests.txt
index 9d266e79c6a2..7eb3cf91c89e 100644
--- a/tools/testing/selftests/lkdtm/tests.txt
+++ b/tools/testing/selftests/lkdtm/tests.txt
@@ -70,3 +70,4 @@ USERCOPY_KERNEL
USERCOPY_KERNEL_DS
STACKLEAK_ERASING OK: the rest of the thread stack is properly erased
CFI_FORWARD_PROTO
+PPC_SLB_MULTIHIT Recovered
--
2.26.2
^ permalink raw reply related
* [PATCH v4 1/2] powerpc/mce: remove nmi_enter/exit from real mode handler
From: Ganesh Goudar @ 2020-10-09 6:40 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: msuchanek, Ganesh Goudar, keescook, npiggin, mahesh
In-Reply-To: <20201009064005.19777-1-ganeshgr@linux.ibm.com>
Use of nmi_enter/exit in real mode handler causes the kernel to panic
and reboot on injecting slb mutihit on pseries machine running in hash
mmu mode, As these calls try to accesses memory outside RMO region in
real mode handler where translation is disabled.
Add check to not to use these calls on pseries machine running in hash
mmu mode.
Fixes: 116ac378bb3f ("powerpc/64s: machine check interrupt update NMI accounting")
Signed-off-by: Ganesh Goudar <ganeshgr@linux.ibm.com>
---
arch/powerpc/kernel/mce.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/kernel/mce.c b/arch/powerpc/kernel/mce.c
index ada59f6c4298..63702c0badb9 100644
--- a/arch/powerpc/kernel/mce.c
+++ b/arch/powerpc/kernel/mce.c
@@ -591,12 +591,11 @@ EXPORT_SYMBOL_GPL(machine_check_print_event_info);
long notrace machine_check_early(struct pt_regs *regs)
{
long handled = 0;
- bool nested = in_nmi();
u8 ftrace_enabled = this_cpu_get_ftrace_enabled();
this_cpu_set_ftrace_enabled(0);
-
- if (!nested)
+ /* Do not use nmi_enter/exit for pseries hpte guest */
+ if (radix_enabled() || !firmware_has_feature(FW_FEATURE_LPAR))
nmi_enter();
hv_nmi_check_nonrecoverable(regs);
@@ -607,7 +606,7 @@ long notrace machine_check_early(struct pt_regs *regs)
if (ppc_md.machine_check_early)
handled = ppc_md.machine_check_early(regs);
- if (!nested)
+ if (radix_enabled() || !firmware_has_feature(FW_FEATURE_LPAR))
nmi_exit();
this_cpu_set_ftrace_enabled(ftrace_enabled);
--
2.26.2
^ permalink raw reply related
* [PATCH v4 0/2] powerpc/mce: Fix mce handler and add selftest
From: Ganesh Goudar @ 2020-10-09 6:40 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: msuchanek, Ganesh Goudar, keescook, npiggin, mahesh
This patch series fixes mce handling for pseries, Adds LKDTM test
for SLB multihit recovery and enables selftest for the same,
basically to test MCE handling on pseries/powernv machines running
in hash mmu mode.
v4:
* Use radix_enabled() to check if its in Hash or Radix mode.
* Use FW_FEATURE_LPAR instead of machine_is_pseries().
v3:
* Merging selftest changes with patch 2/2, Instead of having separate
patch.
* Minor improvements like adding enough comments, Makefile changes,
including header file and adding some prints.
v2:
* Remove in_nmi check before calling nmi_enter/exit,
as nesting is supported.
* Fix build errors and remove unused variables.
* Integrate error injection code into LKDTM.
* Add support to inject multihit in paca.
Ganesh Goudar (2):
powerpc/mce: remove nmi_enter/exit from real mode handler
lkdtm/powerpc: Add SLB multihit test
arch/powerpc/kernel/mce.c | 7 +-
drivers/misc/lkdtm/Makefile | 1 +
drivers/misc/lkdtm/core.c | 3 +
drivers/misc/lkdtm/lkdtm.h | 3 +
drivers/misc/lkdtm/powerpc.c | 156 ++++++++++++++++++++++++
tools/testing/selftests/lkdtm/tests.txt | 1 +
6 files changed, 167 insertions(+), 4 deletions(-)
create mode 100644 drivers/misc/lkdtm/powerpc.c
--
2.26.2
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox