* [PATCH 06/18] io_uring: add fsync support
From: Jens Axboe @ 2019-02-07 19:55 UTC (permalink / raw)
To: linux-aio, linux-block, linux-api
Cc: hch, jmoyer, avi, jannh, viro, Jens Axboe
In-Reply-To: <20190207195552.22770-1-axboe@kernel.dk>
From: Christoph Hellwig <hch@lst.de>
Add a new fsync opcode, which either syncs a range if one is passed,
or the whole file if the offset and length fields are both cleared
to zero. A flag is provided to use fdatasync semantics, that is only
force out metadata which is required to retrieve the file data, but
not others like metadata.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
fs/io_uring.c | 40 +++++++++++++++++++++++++++++++++++
include/uapi/linux/io_uring.h | 8 ++++++-
2 files changed, 47 insertions(+), 1 deletion(-)
diff --git a/fs/io_uring.c b/fs/io_uring.c
index 39a6cd14f0ec..e92ae7abffce 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -4,6 +4,7 @@
* supporting fast/efficient IO.
*
* Copyright (C) 2018-2019 Jens Axboe
+ * Copyright (c) 2018-2019 Christoph Hellwig
*/
#include <linux/kernel.h>
#include <linux/init.h>
@@ -517,6 +518,42 @@ static int io_nop(struct io_kiocb *req, u64 user_data)
return 0;
}
+static int io_fsync(struct io_kiocb *req, const struct io_uring_sqe *sqe,
+ bool force_nonblock)
+{
+ struct io_ring_ctx *ctx = req->ctx;
+ loff_t sqe_off = READ_ONCE(sqe->off);
+ loff_t sqe_len = READ_ONCE(sqe->len);
+ loff_t end = sqe_off + sqe_len;
+ unsigned fsync_flags;
+ struct file *file;
+ int ret, fd;
+
+ /* fsync always requires a blocking context */
+ if (force_nonblock)
+ return -EAGAIN;
+
+ if (unlikely(sqe->addr || sqe->ioprio))
+ return -EINVAL;
+
+ fsync_flags = READ_ONCE(sqe->fsync_flags);
+ if (unlikely(fsync_flags & ~IORING_FSYNC_DATASYNC))
+ return -EINVAL;
+
+ fd = READ_ONCE(sqe->fd);
+ file = fget(fd);
+ if (unlikely(!file))
+ return -EBADF;
+
+ ret = vfs_fsync_range(file, sqe_off, end > 0 ? end : LLONG_MAX,
+ fsync_flags & IORING_FSYNC_DATASYNC);
+
+ fput(file);
+ io_cqring_add_event(ctx, sqe->user_data, ret, 0);
+ io_free_req(req);
+ return 0;
+}
+
static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,
const struct sqe_submit *s, bool force_nonblock)
{
@@ -538,6 +575,9 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,
case IORING_OP_WRITEV:
ret = io_write(req, s, force_nonblock);
break;
+ case IORING_OP_FSYNC:
+ ret = io_fsync(req, s->sqe, force_nonblock);
+ break;
default:
ret = -EINVAL;
break;
diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h
index ac692823d6f4..4589d56d0b68 100644
--- a/include/uapi/linux/io_uring.h
+++ b/include/uapi/linux/io_uring.h
@@ -24,7 +24,7 @@ struct io_uring_sqe {
__u32 len; /* buffer size or number of iovecs */
union {
__kernel_rwf_t rw_flags;
- __u32 __resv;
+ __u32 fsync_flags;
};
__u64 user_data; /* data to be passed back at completion time */
__u64 __pad2[3];
@@ -33,6 +33,12 @@ struct io_uring_sqe {
#define IORING_OP_NOP 0
#define IORING_OP_READV 1
#define IORING_OP_WRITEV 2
+#define IORING_OP_FSYNC 3
+
+/*
+ * sqe->fsync_flags
+ */
+#define IORING_FSYNC_DATASYNC (1U << 0)
/*
* IO completion data structure (Completion Queue Entry)
--
2.17.1
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply related
* [PATCH 05/18] Add io_uring IO interface
From: Jens Axboe @ 2019-02-07 19:55 UTC (permalink / raw)
To: linux-aio, linux-block, linux-api
Cc: hch, jmoyer, avi, jannh, viro, Jens Axboe
In-Reply-To: <20190207195552.22770-1-axboe@kernel.dk>
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_sqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up a context for doing async IO. On success, returns a file
descriptor that the application can mmap to gain access to the
SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
arch/x86/entry/syscalls/syscall_32.tbl | 2 +
arch/x86/entry/syscalls/syscall_64.tbl | 2 +
fs/Makefile | 1 +
fs/io_uring.c | 1174 ++++++++++++++++++++++++
include/linux/fs.h | 9 +
include/linux/syscalls.h | 6 +
include/uapi/asm-generic/unistd.h | 6 +-
include/uapi/linux/io_uring.h | 95 ++
init/Kconfig | 9 +
kernel/sys_ni.c | 2 +
net/unix/garbage.c | 3 +
11 files changed, 1308 insertions(+), 1 deletion(-)
create mode 100644 fs/io_uring.c
create mode 100644 include/uapi/linux/io_uring.h
diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl
index 3cf7b533b3d1..481c126259e9 100644
--- a/arch/x86/entry/syscalls/syscall_32.tbl
+++ b/arch/x86/entry/syscalls/syscall_32.tbl
@@ -398,3 +398,5 @@
384 i386 arch_prctl sys_arch_prctl __ia32_compat_sys_arch_prctl
385 i386 io_pgetevents sys_io_pgetevents __ia32_compat_sys_io_pgetevents
386 i386 rseq sys_rseq __ia32_sys_rseq
+425 i386 io_uring_setup sys_io_uring_setup __ia32_sys_io_uring_setup
+426 i386 io_uring_enter sys_io_uring_enter __ia32_sys_io_uring_enter
diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl
index f0b1709a5ffb..6a32a430c8e0 100644
--- a/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/arch/x86/entry/syscalls/syscall_64.tbl
@@ -343,6 +343,8 @@
332 common statx __x64_sys_statx
333 common io_pgetevents __x64_sys_io_pgetevents
334 common rseq __x64_sys_rseq
+425 common io_uring_setup __x64_sys_io_uring_setup
+426 common io_uring_enter __x64_sys_io_uring_enter
#
# x32-specific system call numbers start at 512 to avoid cache impact
diff --git a/fs/Makefile b/fs/Makefile
index 293733f61594..8e15d6fc4340 100644
--- a/fs/Makefile
+++ b/fs/Makefile
@@ -30,6 +30,7 @@ obj-$(CONFIG_TIMERFD) += timerfd.o
obj-$(CONFIG_EVENTFD) += eventfd.o
obj-$(CONFIG_USERFAULTFD) += userfaultfd.o
obj-$(CONFIG_AIO) += aio.o
+obj-$(CONFIG_IO_URING) += io_uring.o
obj-$(CONFIG_FS_DAX) += dax.o
obj-$(CONFIG_FS_ENCRYPTION) += crypto/
obj-$(CONFIG_FILE_LOCKING) += locks.o
diff --git a/fs/io_uring.c b/fs/io_uring.c
new file mode 100644
index 000000000000..39a6cd14f0ec
--- /dev/null
+++ b/fs/io_uring.c
@@ -0,0 +1,1174 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Shared application/kernel submission and completion ring pairs, for
+ * supporting fast/efficient IO.
+ *
+ * Copyright (C) 2018-2019 Jens Axboe
+ */
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/errno.h>
+#include <linux/syscalls.h>
+#include <linux/compat.h>
+#include <linux/refcount.h>
+#include <linux/uio.h>
+
+#include <linux/sched/signal.h>
+#include <linux/fs.h>
+#include <linux/file.h>
+#include <linux/fdtable.h>
+#include <linux/mm.h>
+#include <linux/mman.h>
+#include <linux/mmu_context.h>
+#include <linux/percpu.h>
+#include <linux/slab.h>
+#include <linux/workqueue.h>
+#include <linux/blkdev.h>
+#include <linux/net.h>
+#include <net/sock.h>
+#include <net/af_unix.h>
+#include <linux/anon_inodes.h>
+#include <linux/sched/mm.h>
+#include <linux/uaccess.h>
+#include <linux/nospec.h>
+
+#include <uapi/linux/io_uring.h>
+
+#include "internal.h"
+
+#define IORING_MAX_ENTRIES 4096
+
+struct io_uring {
+ u32 head ____cacheline_aligned_in_smp;
+ u32 tail ____cacheline_aligned_in_smp;
+};
+
+struct io_sq_ring {
+ struct io_uring r;
+ u32 ring_mask;
+ u32 ring_entries;
+ u32 dropped;
+ u32 flags;
+ u32 array[];
+};
+
+struct io_cq_ring {
+ struct io_uring r;
+ u32 ring_mask;
+ u32 ring_entries;
+ u32 overflow;
+ struct io_uring_cqe cqes[];
+};
+
+struct io_ring_ctx {
+ struct {
+ struct percpu_ref refs;
+ } ____cacheline_aligned_in_smp;
+
+ struct {
+ unsigned int flags;
+ bool compat;
+
+ /* SQ ring */
+ struct io_sq_ring *sq_ring;
+ unsigned cached_sq_head;
+ unsigned sq_entries;
+ unsigned sq_mask;
+ struct io_uring_sqe *sq_sqes;
+ } ____cacheline_aligned_in_smp;
+
+ /* IO offload */
+ struct workqueue_struct *sqo_wq;
+ struct mm_struct *sqo_mm;
+
+ struct {
+ /* CQ ring */
+ struct io_cq_ring *cq_ring;
+ unsigned cached_cq_tail;
+ unsigned cq_entries;
+ unsigned cq_mask;
+ struct wait_queue_head cq_wait;
+ struct fasync_struct *cq_fasync;
+ } ____cacheline_aligned_in_smp;
+
+ struct user_struct *user;
+
+ struct completion ctx_done;
+
+ struct {
+ struct mutex uring_lock;
+ wait_queue_head_t wait;
+ } ____cacheline_aligned_in_smp;
+
+ struct {
+ spinlock_t completion_lock;
+ } ____cacheline_aligned_in_smp;
+
+#if defined(CONFIG_NET)
+ struct socket *ring_sock;
+#endif
+};
+
+struct sqe_submit {
+ const struct io_uring_sqe *sqe;
+ unsigned short index;
+ bool has_user;
+};
+
+struct io_kiocb {
+ struct kiocb rw;
+
+ struct sqe_submit submit;
+
+ struct io_ring_ctx *ctx;
+ struct list_head list;
+ unsigned int flags;
+#define REQ_F_FORCE_NONBLOCK 1 /* inline submission attempt */
+ u64 user_data;
+
+ struct work_struct work;
+};
+
+#define IO_PLUG_THRESHOLD 2
+
+static struct kmem_cache *req_cachep;
+
+static const struct file_operations io_uring_fops;
+
+struct sock *io_uring_get_socket(struct file *file)
+{
+#if defined(CONFIG_NET)
+ if (file->f_op == &io_uring_fops) {
+ struct io_ring_ctx *ctx = file->private_data;
+
+ return ctx->ring_sock->sk;
+ }
+#endif
+ return NULL;
+}
+
+static void io_ring_ctx_ref_free(struct percpu_ref *ref)
+{
+ struct io_ring_ctx *ctx = container_of(ref, struct io_ring_ctx, refs);
+
+ complete(&ctx->ctx_done);
+}
+
+static struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)
+{
+ struct io_ring_ctx *ctx;
+
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return NULL;
+
+ if (percpu_ref_init(&ctx->refs, io_ring_ctx_ref_free, 0, GFP_KERNEL)) {
+ kfree(ctx);
+ return NULL;
+ }
+
+ ctx->flags = p->flags;
+ init_waitqueue_head(&ctx->cq_wait);
+ init_completion(&ctx->ctx_done);
+ mutex_init(&ctx->uring_lock);
+ init_waitqueue_head(&ctx->wait);
+ spin_lock_init(&ctx->completion_lock);
+ return ctx;
+}
+
+static void io_commit_cqring(struct io_ring_ctx *ctx)
+{
+ struct io_cq_ring *ring = ctx->cq_ring;
+
+ if (ctx->cached_cq_tail != ring->r.tail) {
+ /* order cqe stores with ring update */
+ smp_wmb();
+ WRITE_ONCE(ring->r.tail, ctx->cached_cq_tail);
+ /* write side barrier of tail update, app has read side */
+ smp_wmb();
+
+ if (wq_has_sleeper(&ctx->cq_wait)) {
+ wake_up_interruptible(&ctx->cq_wait);
+ kill_fasync(&ctx->cq_fasync, SIGIO, POLL_IN);
+ }
+ }
+}
+
+static struct io_uring_cqe *io_get_cqring(struct io_ring_ctx *ctx)
+{
+ struct io_cq_ring *ring = ctx->cq_ring;
+ unsigned tail;
+
+ tail = ctx->cached_cq_tail;
+ smp_rmb();
+ if (tail + 1 == READ_ONCE(ring->r.head))
+ return NULL;
+
+ ctx->cached_cq_tail++;
+ return &ring->cqes[tail & ctx->cq_mask];
+}
+
+static void io_cqring_fill_event(struct io_ring_ctx *ctx, u64 ki_user_data,
+ long res, unsigned ev_flags)
+{
+ struct io_uring_cqe *cqe;
+
+ /*
+ * If we can't get a cq entry, userspace overflowed the
+ * submission (by quite a lot). Increment the overflow count in
+ * the ring.
+ */
+ cqe = io_get_cqring(ctx);
+ if (cqe) {
+ cqe->user_data = ki_user_data;
+ cqe->res = res;
+ cqe->flags = ev_flags;
+ } else
+ ctx->cq_ring->overflow++;
+}
+
+static void io_cqring_add_event(struct io_ring_ctx *ctx, u64 ki_user_data,
+ long res, unsigned ev_flags)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&ctx->completion_lock, flags);
+ io_cqring_fill_event(ctx, ki_user_data, res, ev_flags);
+ io_commit_cqring(ctx);
+ spin_unlock_irqrestore(&ctx->completion_lock, flags);
+
+ if (waitqueue_active(&ctx->wait))
+ wake_up(&ctx->wait);
+}
+
+static void io_ring_drop_ctx_refs(struct io_ring_ctx *ctx, unsigned refs)
+{
+ percpu_ref_put_many(&ctx->refs, refs);
+
+ if (waitqueue_active(&ctx->wait))
+ wake_up(&ctx->wait);
+}
+
+static struct io_kiocb *io_get_req(struct io_ring_ctx *ctx)
+{
+ struct io_kiocb *req;
+
+ if (!percpu_ref_tryget(&ctx->refs))
+ return NULL;
+
+ req = kmem_cache_alloc(req_cachep, __GFP_NOWARN);
+ if (req) {
+ req->ctx = ctx;
+ req->flags = 0;
+ return req;
+ }
+
+ io_ring_drop_ctx_refs(ctx, 1);
+ return NULL;
+}
+
+static void io_free_req(struct io_kiocb *req)
+{
+ io_ring_drop_ctx_refs(req->ctx, 1);
+ kmem_cache_free(req_cachep, req);
+}
+
+static void kiocb_end_write(struct kiocb *kiocb)
+{
+ if (kiocb->ki_flags & IOCB_WRITE) {
+ struct inode *inode = file_inode(kiocb->ki_filp);
+
+ /*
+ * Tell lockdep we inherited freeze protection from submission
+ * thread.
+ */
+ if (S_ISREG(inode->i_mode))
+ __sb_writers_acquired(inode->i_sb, SB_FREEZE_WRITE);
+ file_end_write(kiocb->ki_filp);
+ }
+}
+
+static void io_complete_rw(struct kiocb *kiocb, long res, long res2)
+{
+ struct io_kiocb *req = container_of(kiocb, struct io_kiocb, rw);
+
+ kiocb_end_write(kiocb);
+
+ fput(kiocb->ki_filp);
+ io_cqring_add_event(req->ctx, req->user_data, res, 0);
+ io_free_req(req);
+}
+
+/*
+ * If we tracked the file through the SCM inflight mechanism, we could support
+ * any file. For now, just ensure that anything potentially problematic is done
+ * inline.
+ */
+static bool io_file_supports_async(struct file *file)
+{
+ umode_t mode = file_inode(file)->i_mode;
+
+ if (S_ISBLK(mode) || S_ISCHR(mode))
+ return true;
+ if (S_ISREG(mode) && file->f_op != &io_uring_fops)
+ return true;
+
+ return false;
+}
+
+static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe,
+ bool force_nonblock)
+{
+ struct kiocb *kiocb = &req->rw;
+ unsigned ioprio;
+ int fd, ret;
+
+ /* For -EAGAIN retry, everything is already prepped */
+ if (kiocb->ki_filp)
+ return 0;
+
+ fd = READ_ONCE(sqe->fd);
+ kiocb->ki_filp = fget(fd);
+ if (unlikely(!kiocb->ki_filp))
+ return -EBADF;
+ if (force_nonblock && !io_file_supports_async(kiocb->ki_filp))
+ force_nonblock = false;
+ kiocb->ki_pos = READ_ONCE(sqe->off);
+ kiocb->ki_flags = iocb_flags(kiocb->ki_filp);
+ kiocb->ki_hint = ki_hint_validate(file_write_hint(kiocb->ki_filp));
+
+ ioprio = READ_ONCE(sqe->ioprio);
+ if (ioprio) {
+ ret = ioprio_check_cap(ioprio);
+ if (ret)
+ goto out_fput;
+
+ kiocb->ki_ioprio = ioprio;
+ } else
+ kiocb->ki_ioprio = get_current_ioprio();
+
+ ret = kiocb_set_rw_flags(kiocb, READ_ONCE(sqe->rw_flags));
+ if (unlikely(ret))
+ goto out_fput;
+ if (force_nonblock) {
+ kiocb->ki_flags |= IOCB_NOWAIT;
+ req->flags |= REQ_F_FORCE_NONBLOCK;
+ }
+ if (kiocb->ki_flags & IOCB_HIPRI) {
+ ret = -EINVAL;
+ goto out_fput;
+ }
+
+ kiocb->ki_complete = io_complete_rw;
+ return 0;
+out_fput:
+ fput(kiocb->ki_filp);
+ return ret;
+}
+
+static inline void io_rw_done(struct kiocb *kiocb, ssize_t ret)
+{
+ switch (ret) {
+ case -EIOCBQUEUED:
+ break;
+ case -ERESTARTSYS:
+ case -ERESTARTNOINTR:
+ case -ERESTARTNOHAND:
+ case -ERESTART_RESTARTBLOCK:
+ /*
+ * We can't just restart the syscall, since previously
+ * submitted sqes may already be in progress. Just fail this
+ * IO with EINTR.
+ */
+ ret = -EINTR;
+ /* fall through */
+ default:
+ kiocb->ki_complete(kiocb, ret, 0);
+ }
+}
+
+static int io_import_iovec(struct io_ring_ctx *ctx, int rw,
+ const struct sqe_submit *s, struct iovec **iovec,
+ struct iov_iter *iter)
+{
+ const struct io_uring_sqe *sqe = s->sqe;
+ void __user *buf = u64_to_user_ptr(READ_ONCE(sqe->addr));
+ size_t sqe_len = READ_ONCE(sqe->len);
+
+ if (!s->has_user)
+ return EFAULT;
+
+#ifdef CONFIG_COMPAT
+ if (ctx->compat)
+ return compat_import_iovec(rw, buf, sqe_len, UIO_FASTIOV,
+ iovec, iter);
+#endif
+
+ return import_iovec(rw, buf, sqe_len, UIO_FASTIOV, iovec, iter);
+}
+
+static ssize_t io_read(struct io_kiocb *req, const struct sqe_submit *s,
+ bool force_nonblock)
+{
+ struct iovec inline_vecs[UIO_FASTIOV], *iovec = inline_vecs;
+ struct kiocb *kiocb = &req->rw;
+ struct iov_iter iter;
+ struct file *file;
+ ssize_t ret;
+
+ ret = io_prep_rw(req, s->sqe, force_nonblock);
+ if (ret)
+ return ret;
+ file = kiocb->ki_filp;
+
+ ret = -EBADF;
+ if (unlikely(!(file->f_mode & FMODE_READ)))
+ goto out_fput;
+ ret = -EINVAL;
+ if (unlikely(!file->f_op->read_iter))
+ goto out_fput;
+
+ ret = io_import_iovec(req->ctx, READ, s, &iovec, &iter);
+ if (ret)
+ goto out_fput;
+
+ ret = rw_verify_area(READ, file, &kiocb->ki_pos, iov_iter_count(&iter));
+ if (!ret) {
+ ssize_t ret2;
+
+ /* Catch -EAGAIN return for forced non-blocking submission */
+ ret2 = call_read_iter(file, kiocb, &iter);
+ if (!force_nonblock || ret2 != -EAGAIN)
+ io_rw_done(kiocb, ret2);
+ else
+ ret = -EAGAIN;
+ }
+ kfree(iovec);
+out_fput:
+ /* Hold on to the file for -EAGAIN */
+ if (unlikely(ret && ret != -EAGAIN))
+ fput(file);
+ return ret;
+}
+
+static ssize_t io_write(struct io_kiocb *req, const struct sqe_submit *s,
+ bool force_nonblock)
+{
+ struct iovec inline_vecs[UIO_FASTIOV], *iovec = inline_vecs;
+ struct kiocb *kiocb = &req->rw;
+ struct iov_iter iter;
+ struct file *file;
+ ssize_t ret;
+
+ ret = io_prep_rw(req, s->sqe, force_nonblock);
+ if (ret)
+ return ret;
+ /* Hold on to the file for -EAGAIN */
+ if (force_nonblock && !(kiocb->ki_flags & IOCB_DIRECT))
+ return -EAGAIN;
+
+ ret = -EBADF;
+ file = kiocb->ki_filp;
+ if (unlikely(!(file->f_mode & FMODE_WRITE)))
+ goto out_fput;
+ ret = -EINVAL;
+ if (unlikely(!file->f_op->write_iter))
+ goto out_fput;
+
+ ret = io_import_iovec(req->ctx, WRITE, s, &iovec, &iter);
+ if (ret)
+ goto out_fput;
+
+ ret = rw_verify_area(WRITE, file, &kiocb->ki_pos,
+ iov_iter_count(&iter));
+ if (!ret) {
+ /*
+ * Open-code file_start_write here to grab freeze protection,
+ * which will be released by another thread in
+ * io_complete_rw(). Fool lockdep by telling it the lock got
+ * released so that it doesn't complain about the held lock when
+ * we return to userspace.
+ */
+ if (S_ISREG(file_inode(file)->i_mode)) {
+ __sb_start_write(file_inode(file)->i_sb,
+ SB_FREEZE_WRITE, true);
+ __sb_writers_release(file_inode(file)->i_sb,
+ SB_FREEZE_WRITE);
+ }
+ kiocb->ki_flags |= IOCB_WRITE;
+ io_rw_done(kiocb, call_write_iter(file, kiocb, &iter));
+ }
+ kfree(iovec);
+out_fput:
+ if (unlikely(ret))
+ fput(file);
+ return ret;
+}
+
+/*
+ * IORING_OP_NOP just posts a completion event, nothing else.
+ */
+static int io_nop(struct io_kiocb *req, u64 user_data)
+{
+ struct io_ring_ctx *ctx = req->ctx;
+
+ io_cqring_add_event(ctx, user_data, 0, 0);
+ io_free_req(req);
+ return 0;
+}
+
+static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,
+ const struct sqe_submit *s, bool force_nonblock)
+{
+ ssize_t ret;
+ int opcode;
+
+ if (unlikely(s->index >= ctx->sq_entries))
+ return -EINVAL;
+ req->user_data = READ_ONCE(s->sqe->user_data);
+
+ opcode = READ_ONCE(s->sqe->opcode);
+ switch (opcode) {
+ case IORING_OP_NOP:
+ ret = io_nop(req, req->user_data);
+ break;
+ case IORING_OP_READV:
+ ret = io_read(req, s, force_nonblock);
+ break;
+ case IORING_OP_WRITEV:
+ ret = io_write(req, s, force_nonblock);
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static void io_sq_wq_submit_work(struct work_struct *work)
+{
+ struct io_kiocb *req = container_of(work, struct io_kiocb, work);
+ struct sqe_submit *s = &req->submit;
+ u64 user_data = READ_ONCE(s->sqe->user_data);
+ struct io_ring_ctx *ctx = req->ctx;
+ mm_segment_t old_fs = get_fs();
+ int ret;
+
+ /* Ensure we clear previously set forced non-block flag */
+ req->flags &= ~REQ_F_FORCE_NONBLOCK;
+ req->rw.ki_flags &= ~IOCB_NOWAIT;
+
+ if (!mmget_not_zero(ctx->sqo_mm)) {
+ ret = -EFAULT;
+ goto err;
+ }
+
+ use_mm(ctx->sqo_mm);
+ set_fs(USER_DS);
+ s->has_user = true;
+
+ ret = __io_submit_sqe(ctx, req, s, false);
+
+ set_fs(old_fs);
+ unuse_mm(ctx->sqo_mm);
+ mmput(ctx->sqo_mm);
+err:
+ if (ret) {
+ io_cqring_add_event(ctx, user_data, ret, 0);
+ io_free_req(req);
+ }
+}
+
+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s)
+{
+ struct io_kiocb *req;
+ ssize_t ret;
+
+ /* enforce forwards compatibility on users */
+ if (unlikely(s->sqe->flags))
+ return -EINVAL;
+
+ req = io_get_req(ctx);
+ if (unlikely(!req))
+ return -EAGAIN;
+
+ req->rw.ki_filp = NULL;
+
+ ret = __io_submit_sqe(ctx, req, s, true);
+ if (ret == -EAGAIN) {
+ memcpy(&req->submit, s, sizeof(*s));
+ INIT_WORK(&req->work, io_sq_wq_submit_work);
+ queue_work(ctx->sqo_wq, &req->work);
+ ret = 0;
+ }
+ if (ret)
+ io_free_req(req);
+
+ return ret;
+}
+
+static void io_commit_sqring(struct io_ring_ctx *ctx)
+{
+ struct io_sq_ring *ring = ctx->sq_ring;
+
+ if (ctx->cached_sq_head != ring->r.head) {
+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head);
+ /* write side barrier of head update, app has read side */
+ smp_wmb();
+ }
+}
+
+/*
+ * Undo last io_get_sqring()
+ */
+static void io_drop_sqring(struct io_ring_ctx *ctx)
+{
+ ctx->cached_sq_head--;
+}
+
+/*
+ * Fetch an sqe, if one is available. Note that s->sqe will point to memory
+ * that is mapped by userspace. This means that care needs to be taken to
+ * ensure that reads are stable, as we cannot rely on userspace always
+ * being a good citizen. If members of the sqe are validated and then later
+ * used, it's important that those reads are done through READ_ONCE() to
+ * prevent a re-load down the line.
+ */
+static bool io_get_sqring(struct io_ring_ctx *ctx, struct sqe_submit *s)
+{
+ struct io_sq_ring *ring = ctx->sq_ring;
+ unsigned head;
+
+ /*
+ * The cached sq head (or cq tail) serves two purposes:
+ *
+ * 1) allows us to batch the cost of updating the user visible
+ * head updates.
+ * 2) allows the kernel side to track the head on its own, even
+ * though the application is the one updating it.
+ */
+ head = ctx->cached_sq_head;
+ smp_rmb();
+ if (head == READ_ONCE(ring->r.tail))
+ return false;
+
+ head = READ_ONCE(ring->array[head & ctx->sq_mask]);
+ if (head < ctx->sq_entries) {
+ s->index = head;
+ s->sqe = &ctx->sq_sqes[head];
+ ctx->cached_sq_head++;
+ return true;
+ }
+
+ /* drop invalid entries */
+ ctx->cached_sq_head++;
+ ring->dropped++;
+ smp_wmb();
+ return false;
+}
+
+static int io_ring_submit(struct io_ring_ctx *ctx, unsigned int to_submit)
+{
+ int i, ret = 0, submit = 0;
+ struct blk_plug plug;
+
+ if (to_submit > IO_PLUG_THRESHOLD)
+ blk_start_plug(&plug);
+
+ for (i = 0; i < to_submit; i++) {
+ struct sqe_submit s;
+
+ if (!io_get_sqring(ctx, &s))
+ break;
+
+ s.has_user = true;
+ ret = io_submit_sqe(ctx, &s);
+ if (ret) {
+ io_drop_sqring(ctx);
+ break;
+ }
+
+ submit++;
+ }
+ io_commit_sqring(ctx);
+
+ if (to_submit > IO_PLUG_THRESHOLD)
+ blk_finish_plug(&plug);
+
+ return submit ? submit : ret;
+}
+
+/*
+ * Wait until events become available, if we don't already have some. The
+ * application must reap them itself, as they reside on the shared cq ring.
+ */
+static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events,
+ const sigset_t __user *sig, size_t sigsz)
+{
+ struct io_cq_ring *ring = ctx->cq_ring;
+ sigset_t ksigmask, sigsaved;
+ DEFINE_WAIT(wait);
+ int ret = 0;
+
+ smp_rmb();
+ if (ring->r.head != ring->r.tail)
+ return 0;
+ if (!min_events)
+ return 0;
+
+ if (sig) {
+ ret = set_user_sigmask(sig, &ksigmask, &sigsaved, sigsz);
+ if (ret)
+ return ret;
+ }
+
+ do {
+ prepare_to_wait(&ctx->wait, &wait, TASK_INTERRUPTIBLE);
+
+ ret = 0;
+ smp_rmb();
+ if (ring->r.head != ring->r.tail)
+ break;
+
+ schedule();
+
+ ret = -EINTR;
+ if (signal_pending(current))
+ break;
+ } while (1);
+
+ finish_wait(&ctx->wait, &wait);
+
+ if (sig)
+ restore_user_sigmask(sig, &sigsaved);
+
+ return ring->r.head == ring->r.tail ? ret : 0;
+}
+
+static int io_sq_offload_start(struct io_ring_ctx *ctx)
+{
+ int ret;
+
+ mmgrab(current->mm);
+ ctx->sqo_mm = current->mm;
+
+ /* Do QD, or 2 * CPUS, whatever is smallest */
+ ctx->sqo_wq = alloc_workqueue("io_ring-wq", WQ_UNBOUND | WQ_FREEZABLE,
+ min(ctx->sq_entries - 1, 2 * num_online_cpus()));
+ if (!ctx->sqo_wq) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ return 0;
+err:
+ mmdrop(ctx->sqo_mm);
+ ctx->sqo_mm = NULL;
+ return ret;
+}
+
+static void io_unaccount_mem(struct user_struct *user, unsigned long nr_pages)
+{
+ if (capable(CAP_IPC_LOCK))
+ return;
+ atomic_long_sub(nr_pages, &user->locked_vm);
+}
+
+static int io_account_mem(struct user_struct *user, unsigned long nr_pages)
+{
+ unsigned long page_limit, cur_pages, new_pages;
+
+ if (capable(CAP_IPC_LOCK))
+ return 0;
+
+ /* Don't allow more pages than we can safely lock */
+ page_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+
+ do {
+ cur_pages = atomic_long_read(&user->locked_vm);
+ new_pages = cur_pages + nr_pages;
+ if (new_pages > page_limit)
+ return -ENOMEM;
+ } while (atomic_long_cmpxchg(&user->locked_vm, cur_pages,
+ new_pages) != cur_pages);
+
+ return 0;
+}
+
+static void io_mem_free(void *ptr)
+{
+ struct page *page = virt_to_head_page(ptr);
+
+ if (put_page_testzero(page))
+ free_compound_page(page);
+}
+
+static void *io_mem_alloc(size_t size)
+{
+ gfp_t gfp_flags = GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN | __GFP_COMP |
+ __GFP_NORETRY;
+
+ return (void *) __get_free_pages(gfp_flags, get_order(size));
+}
+
+static unsigned long ring_pages(unsigned sq_entries, unsigned cq_entries)
+{
+ struct io_sq_ring *sq_ring;
+ struct io_cq_ring *cq_ring;
+ size_t bytes;
+
+ bytes = struct_size(sq_ring, array, sq_entries);
+ bytes += array_size(sizeof(struct io_uring_sqe), sq_entries);
+ bytes += struct_size(cq_ring, cqes, cq_entries);
+
+ return (bytes + PAGE_SIZE - 1) / PAGE_SIZE;
+}
+
+static void io_ring_ctx_free(struct io_ring_ctx *ctx)
+{
+ if (ctx->sqo_wq)
+ destroy_workqueue(ctx->sqo_wq);
+ if (ctx->sqo_mm)
+ mmdrop(ctx->sqo_mm);
+#if defined(CONFIG_NET)
+ if (ctx->ring_sock)
+ sock_release(ctx->ring_sock);
+#endif
+
+ io_mem_free(ctx->sq_ring);
+ io_mem_free(ctx->sq_sqes);
+ io_mem_free(ctx->cq_ring);
+
+ percpu_ref_exit(&ctx->refs);
+ io_unaccount_mem(ctx->user,
+ ring_pages(ctx->sq_entries, ctx->cq_entries));
+ free_uid(ctx->user);
+ kfree(ctx);
+}
+
+static __poll_t io_uring_poll(struct file *file, poll_table *wait)
+{
+ struct io_ring_ctx *ctx = file->private_data;
+ __poll_t mask = 0;
+
+ poll_wait(file, &ctx->cq_wait, wait);
+ smp_rmb();
+ if (ctx->sq_ring->r.tail + 1 != ctx->cached_sq_head)
+ mask |= EPOLLOUT | EPOLLWRNORM;
+ if (ctx->cq_ring->r.head != ctx->cached_cq_tail)
+ mask |= EPOLLIN | EPOLLRDNORM;
+
+ return mask;
+}
+
+static int io_uring_fasync(int fd, struct file *file, int on)
+{
+ struct io_ring_ctx *ctx = file->private_data;
+
+ return fasync_helper(fd, file, on, &ctx->cq_fasync);
+}
+
+static void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx)
+{
+ mutex_lock(&ctx->uring_lock);
+ percpu_ref_kill(&ctx->refs);
+ mutex_unlock(&ctx->uring_lock);
+
+ wait_for_completion(&ctx->ctx_done);
+ io_ring_ctx_free(ctx);
+}
+
+static int io_uring_release(struct inode *inode, struct file *file)
+{
+ struct io_ring_ctx *ctx = file->private_data;
+
+ file->private_data = NULL;
+ io_ring_ctx_wait_and_kill(ctx);
+ return 0;
+}
+
+static int io_uring_mmap(struct file *file, struct vm_area_struct *vma)
+{
+ loff_t offset = (loff_t) vma->vm_pgoff << PAGE_SHIFT;
+ unsigned long sz = vma->vm_end - vma->vm_start;
+ struct io_ring_ctx *ctx = file->private_data;
+ unsigned long pfn;
+ struct page *page;
+ void *ptr;
+
+ switch (offset) {
+ case IORING_OFF_SQ_RING:
+ ptr = ctx->sq_ring;
+ break;
+ case IORING_OFF_SQES:
+ ptr = ctx->sq_sqes;
+ break;
+ case IORING_OFF_CQ_RING:
+ ptr = ctx->cq_ring;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ page = virt_to_head_page(ptr);
+ if (sz > (PAGE_SIZE << compound_order(page)))
+ return -EINVAL;
+
+ pfn = virt_to_phys(ptr) >> PAGE_SHIFT;
+ return remap_pfn_range(vma, vma->vm_start, pfn, sz, vma->vm_page_prot);
+}
+
+SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit,
+ u32, min_complete, u32, flags, const sigset_t __user *, sig,
+ size_t, sigsz)
+{
+ struct io_ring_ctx *ctx;
+ long ret = -EBADF;
+ int submitted = 0;
+ struct fd f;
+
+ if (flags & ~IORING_ENTER_GETEVENTS)
+ return -EINVAL;
+
+ f = fdget(fd);
+ if (!f.file)
+ return -EBADF;
+
+ ret = -EOPNOTSUPP;
+ if (f.file->f_op != &io_uring_fops)
+ goto out_fput;
+
+ ret = -ENXIO;
+ ctx = f.file->private_data;
+ if (!percpu_ref_tryget(&ctx->refs))
+ goto out_fput;
+
+ if (to_submit) {
+ to_submit = min(to_submit, ctx->sq_entries);
+
+ mutex_lock(&ctx->uring_lock);
+ submitted = io_ring_submit(ctx, to_submit);
+ mutex_unlock(&ctx->uring_lock);
+
+ if (submitted < 0)
+ goto out_ctx;
+ }
+ if (flags & IORING_ENTER_GETEVENTS) {
+ /*
+ * The application could have included the 'to_submit' count
+ * in how many events it wanted to wait for. If we failed to
+ * submit the desired count, we may need to adjust the number
+ * of events to poll/wait for.
+ */
+ if (submitted < to_submit)
+ min_complete = min_t(unsigned, submitted, min_complete);
+
+ ret = io_cqring_wait(ctx, min_complete, sig, sigsz);
+ }
+
+out_ctx:
+ io_ring_drop_ctx_refs(ctx, 1);
+out_fput:
+ fdput(f);
+ return submitted ? submitted : ret;
+}
+
+static const struct file_operations io_uring_fops = {
+ .release = io_uring_release,
+ .mmap = io_uring_mmap,
+ .poll = io_uring_poll,
+ .fasync = io_uring_fasync,
+};
+
+static int io_allocate_scq_urings(struct io_ring_ctx *ctx,
+ struct io_uring_params *p)
+{
+ struct io_sq_ring *sq_ring;
+ struct io_cq_ring *cq_ring;
+ size_t size;
+
+ sq_ring = io_mem_alloc(struct_size(sq_ring, array, p->sq_entries));
+ if (!sq_ring)
+ return -ENOMEM;
+
+ ctx->sq_ring = sq_ring;
+ sq_ring->ring_mask = p->sq_entries - 1;
+ sq_ring->ring_entries = p->sq_entries;
+ ctx->sq_mask = sq_ring->ring_mask;
+ ctx->sq_entries = sq_ring->ring_entries;
+
+ size = array_size(sizeof(struct io_uring_sqe), p->sq_entries);
+ if (size == SIZE_MAX)
+ return -EOVERFLOW;
+
+ ctx->sq_sqes = io_mem_alloc(size);
+ if (!ctx->sq_sqes) {
+ io_mem_free(ctx->sq_ring);
+ return -ENOMEM;
+ }
+
+ cq_ring = io_mem_alloc(struct_size(cq_ring, cqes, p->cq_entries));
+ if (!cq_ring) {
+ io_mem_free(ctx->sq_ring);
+ io_mem_free(ctx->sq_sqes);
+ return -ENOMEM;
+ }
+
+ ctx->cq_ring = cq_ring;
+ cq_ring->ring_mask = p->cq_entries - 1;
+ cq_ring->ring_entries = p->cq_entries;
+ ctx->cq_mask = cq_ring->ring_mask;
+ ctx->cq_entries = cq_ring->ring_entries;
+ return 0;
+}
+
+static int io_uring_get_fd(struct io_ring_ctx *ctx)
+{
+ struct file *file;
+ int ret;
+
+#if defined(CONFIG_NET)
+ ret = sock_create_kern(&init_net, PF_UNIX, SOCK_RAW, IPPROTO_IP,
+ &ctx->ring_sock);
+ if (ret)
+ return ret;
+#endif
+
+ ret = get_unused_fd_flags(O_RDWR | O_CLOEXEC);
+ if (ret < 0)
+ goto err;
+
+ file = anon_inode_getfile("[io_uring]", &io_uring_fops, ctx,
+ O_RDWR | O_CLOEXEC);
+ if (IS_ERR(file)) {
+ put_unused_fd(ret);
+ ret = PTR_ERR(file);
+ goto err;
+ }
+
+#if defined(CONFIG_NET)
+ ctx->ring_sock->file = file;
+#endif
+ fd_install(ret, file);
+ return ret;
+err:
+#if defined(CONFIG_NET)
+ sock_release(ctx->ring_sock);
+ ctx->ring_sock = NULL;
+#endif
+ return ret;
+}
+
+static int io_uring_create(unsigned entries, struct io_uring_params *p)
+{
+ struct user_struct *user = NULL;
+ struct io_ring_ctx *ctx;
+ int ret;
+
+ if (!entries || entries > IORING_MAX_ENTRIES)
+ return -EINVAL;
+
+ /*
+ * Use twice as many entries for the CQ ring. It's possible for the
+ * application to drive a higher depth than the size of the SQ ring,
+ * since the sqes are only used at submission time. This allows for
+ * some flexibility in overcommitting a bit.
+ */
+ p->sq_entries = roundup_pow_of_two(entries);
+ p->cq_entries = 2 * p->sq_entries;
+
+ user = get_uid(current_user());
+ ret = io_account_mem(user, ring_pages(p->sq_entries, p->cq_entries));
+ if (ret) {
+ free_uid(user);
+ return ret;
+ }
+
+ ctx = io_ring_ctx_alloc(p);
+ if (!ctx) {
+ io_unaccount_mem(user, ring_pages(p->sq_entries,
+ p->cq_entries));
+ free_uid(user);
+ return -ENOMEM;
+ }
+ ctx->compat = in_compat_syscall();
+ ctx->user = user;
+
+ ret = io_allocate_scq_urings(ctx, p);
+ if (ret)
+ goto err;
+
+ ret = io_sq_offload_start(ctx);
+ if (ret)
+ goto err;
+
+ ret = io_uring_get_fd(ctx);
+ if (ret < 0)
+ goto err;
+
+ memset(&p->sq_off, 0, sizeof(p->sq_off));
+ p->sq_off.head = offsetof(struct io_sq_ring, r.head);
+ p->sq_off.tail = offsetof(struct io_sq_ring, r.tail);
+ p->sq_off.ring_mask = offsetof(struct io_sq_ring, ring_mask);
+ p->sq_off.ring_entries = offsetof(struct io_sq_ring, ring_entries);
+ p->sq_off.flags = offsetof(struct io_sq_ring, flags);
+ p->sq_off.dropped = offsetof(struct io_sq_ring, dropped);
+ p->sq_off.array = offsetof(struct io_sq_ring, array);
+
+ memset(&p->cq_off, 0, sizeof(p->cq_off));
+ p->cq_off.head = offsetof(struct io_cq_ring, r.head);
+ p->cq_off.tail = offsetof(struct io_cq_ring, r.tail);
+ p->cq_off.ring_mask = offsetof(struct io_cq_ring, ring_mask);
+ p->cq_off.ring_entries = offsetof(struct io_cq_ring, ring_entries);
+ p->cq_off.overflow = offsetof(struct io_cq_ring, overflow);
+ p->cq_off.cqes = offsetof(struct io_cq_ring, cqes);
+ return ret;
+err:
+ io_ring_ctx_wait_and_kill(ctx);
+ return ret;
+}
+
+/*
+ * Sets up an aio uring context, and returns the fd. Applications asks for a
+ * ring size, we return the actual sq/cq ring sizes (among other things) in the
+ * params structure passed in.
+ */
+static long io_uring_setup(u32 entries, struct io_uring_params __user *params)
+{
+ struct io_uring_params p;
+ long ret;
+ int i;
+
+ if (copy_from_user(&p, params, sizeof(p)))
+ return -EFAULT;
+ for (i = 0; i < ARRAY_SIZE(p.resv); i++) {
+ if (p.resv[i])
+ return -EINVAL;
+ }
+
+ if (p.flags)
+ return -EINVAL;
+
+ ret = io_uring_create(entries, &p);
+ if (ret < 0)
+ return ret;
+
+ if (copy_to_user(params, &p, sizeof(p)))
+ return -EFAULT;
+
+ return ret;
+}
+
+SYSCALL_DEFINE2(io_uring_setup, u32, entries,
+ struct io_uring_params __user *, params)
+{
+ return io_uring_setup(entries, params);
+}
+
+static int __init io_uring_init(void)
+{
+ req_cachep = KMEM_CACHE(io_kiocb, SLAB_HWCACHE_ALIGN | SLAB_PANIC);
+ return 0;
+};
+__initcall(io_uring_init);
diff --git a/include/linux/fs.h b/include/linux/fs.h
index dedcc2e9265c..61aa210f0c2b 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -3517,4 +3517,13 @@ extern void inode_nohighmem(struct inode *inode);
extern int vfs_fadvise(struct file *file, loff_t offset, loff_t len,
int advice);
+#if defined(CONFIG_IO_URING)
+extern struct sock *io_uring_get_socket(struct file *file);
+#else
+static inline struct sock *io_uring_get_socket(struct file *file)
+{
+ return NULL;
+}
+#endif
+
#endif /* _LINUX_FS_H */
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index 257cccba3062..3072dbaa7869 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -69,6 +69,7 @@ struct file_handle;
struct sigaltstack;
struct rseq;
union bpf_attr;
+struct io_uring_params;
#include <linux/types.h>
#include <linux/aio_abi.h>
@@ -309,6 +310,11 @@ asmlinkage long sys_io_pgetevents_time32(aio_context_t ctx_id,
struct io_event __user *events,
struct old_timespec32 __user *timeout,
const struct __aio_sigset *sig);
+asmlinkage long sys_io_uring_setup(u32 entries,
+ struct io_uring_params __user *p);
+asmlinkage long sys_io_uring_enter(unsigned int fd, u32 to_submit,
+ u32 min_complete, u32 flags,
+ const sigset_t __user *sig, size_t sigsz);
/* fs/xattr.c */
asmlinkage long sys_setxattr(const char __user *path, const char __user *name,
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index d90127298f12..87871e7b7ea7 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -740,9 +740,13 @@ __SC_COMP(__NR_io_pgetevents, sys_io_pgetevents, compat_sys_io_pgetevents)
__SYSCALL(__NR_rseq, sys_rseq)
#define __NR_kexec_file_load 294
__SYSCALL(__NR_kexec_file_load, sys_kexec_file_load)
+#define __NR_io_uring_setup 425
+__SYSCALL(__NR_io_uring_setup, sys_io_uring_setup)
+#define __NR_io_uring_enter 426
+__SYSCALL(__NR_io_uring_enter, sys_io_uring_enter)
#undef __NR_syscalls
-#define __NR_syscalls 295
+#define __NR_syscalls 427
/*
* 32 bit systems traditionally used different
diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h
new file mode 100644
index 000000000000..ac692823d6f4
--- /dev/null
+++ b/include/uapi/linux/io_uring.h
@@ -0,0 +1,95 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Header file for the io_uring interface.
+ *
+ * Copyright (C) 2019 Jens Axboe
+ * Copyright (C) 2019 Christoph Hellwig
+ */
+#ifndef LINUX_IO_URING_H
+#define LINUX_IO_URING_H
+
+#include <linux/fs.h>
+#include <linux/types.h>
+
+/*
+ * IO submission data structure (Submission Queue Entry)
+ */
+struct io_uring_sqe {
+ __u8 opcode; /* type of operation for this sqe */
+ __u8 flags; /* as of now unused */
+ __u16 ioprio; /* ioprio for the request */
+ __s32 fd; /* file descriptor to do IO on */
+ __u64 off; /* offset into file */
+ __u64 addr; /* pointer to buffer or iovecs */
+ __u32 len; /* buffer size or number of iovecs */
+ union {
+ __kernel_rwf_t rw_flags;
+ __u32 __resv;
+ };
+ __u64 user_data; /* data to be passed back at completion time */
+ __u64 __pad2[3];
+};
+
+#define IORING_OP_NOP 0
+#define IORING_OP_READV 1
+#define IORING_OP_WRITEV 2
+
+/*
+ * IO completion data structure (Completion Queue Entry)
+ */
+struct io_uring_cqe {
+ __u64 user_data; /* sqe->data submission passed back */
+ __s32 res; /* result code for this event */
+ __u32 flags;
+};
+
+/*
+ * Magic offsets for the application to mmap the data it needs
+ */
+#define IORING_OFF_SQ_RING 0ULL
+#define IORING_OFF_CQ_RING 0x8000000ULL
+#define IORING_OFF_SQES 0x10000000ULL
+
+/*
+ * Filled with the offset for mmap(2)
+ */
+struct io_sqring_offsets {
+ __u32 head;
+ __u32 tail;
+ __u32 ring_mask;
+ __u32 ring_entries;
+ __u32 flags;
+ __u32 dropped;
+ __u32 array;
+ __u32 resv1;
+ __u64 resv2;
+};
+
+struct io_cqring_offsets {
+ __u32 head;
+ __u32 tail;
+ __u32 ring_mask;
+ __u32 ring_entries;
+ __u32 overflow;
+ __u32 cqes;
+ __u64 resv[2];
+};
+
+/*
+ * io_uring_enter(2) flags
+ */
+#define IORING_ENTER_GETEVENTS (1U << 0)
+
+/*
+ * Passed in for io_uring_setup(2). Copied back with updated info on success
+ */
+struct io_uring_params {
+ __u32 sq_entries;
+ __u32 cq_entries;
+ __u32 flags;
+ __u32 resv[7];
+ struct io_sqring_offsets sq_off;
+ struct io_cqring_offsets cq_off;
+};
+
+#endif
diff --git a/init/Kconfig b/init/Kconfig
index c9386a365eea..8b5e0da04384 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1414,6 +1414,15 @@ config AIO
by some high performance threaded applications. Disabling
this option saves about 7k.
+config IO_URING
+ bool "Enable IO uring support" if EXPERT
+ select ANON_INODES
+ default y
+ help
+ This option enables support for the io_uring interface, enabling
+ applications to submit and completion IO through submission and
+ completion rings that are shared between the kernel and application.
+
config ADVISE_SYSCALLS
bool "Enable madvise/fadvise syscalls" if EXPERT
default y
diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c
index ab9d0e3c6d50..ee5e523564bb 100644
--- a/kernel/sys_ni.c
+++ b/kernel/sys_ni.c
@@ -46,6 +46,8 @@ COND_SYSCALL(io_getevents);
COND_SYSCALL(io_pgetevents);
COND_SYSCALL_COMPAT(io_getevents);
COND_SYSCALL_COMPAT(io_pgetevents);
+COND_SYSCALL(io_uring_setup);
+COND_SYSCALL(io_uring_enter);
/* fs/xattr.c */
diff --git a/net/unix/garbage.c b/net/unix/garbage.c
index c36757e72844..f81854d74c7d 100644
--- a/net/unix/garbage.c
+++ b/net/unix/garbage.c
@@ -108,6 +108,9 @@ struct sock *unix_get_socket(struct file *filp)
/* PF_UNIX ? */
if (s && sock->ops && sock->ops->family == PF_UNIX)
u_sock = s;
+ } else {
+ /* Could be an io_uring instance */
+ u_sock = io_uring_get_socket(filp);
}
return u_sock;
}
--
2.17.1
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply related
* [PATCH 04/18] iomap: wire up the iopoll method
From: Jens Axboe @ 2019-02-07 19:55 UTC (permalink / raw)
To: linux-aio, linux-block, linux-api
Cc: hch, jmoyer, avi, jannh, viro, Jens Axboe
In-Reply-To: <20190207195552.22770-1-axboe@kernel.dk>
From: Christoph Hellwig <hch@lst.de>
Store the request queue the last bio was submitted to in the iocb
private data in addition to the cookie so that we find the right block
device. Also refactor the common direct I/O bio submission code into a
nice little helper.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Modified to use bio_set_polled().
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
fs/gfs2/file.c | 2 ++
fs/iomap.c | 43 ++++++++++++++++++++++++++++---------------
fs/xfs/xfs_file.c | 1 +
include/linux/iomap.h | 1 +
4 files changed, 32 insertions(+), 15 deletions(-)
diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
index a2dea5bc0427..58a768e59712 100644
--- a/fs/gfs2/file.c
+++ b/fs/gfs2/file.c
@@ -1280,6 +1280,7 @@ const struct file_operations gfs2_file_fops = {
.llseek = gfs2_llseek,
.read_iter = gfs2_file_read_iter,
.write_iter = gfs2_file_write_iter,
+ .iopoll = iomap_dio_iopoll,
.unlocked_ioctl = gfs2_ioctl,
.mmap = gfs2_mmap,
.open = gfs2_open,
@@ -1310,6 +1311,7 @@ const struct file_operations gfs2_file_fops_nolock = {
.llseek = gfs2_llseek,
.read_iter = gfs2_file_read_iter,
.write_iter = gfs2_file_write_iter,
+ .iopoll = iomap_dio_iopoll,
.unlocked_ioctl = gfs2_ioctl,
.mmap = gfs2_mmap,
.open = gfs2_open,
diff --git a/fs/iomap.c b/fs/iomap.c
index 897c60215dd1..2ac9eb746d44 100644
--- a/fs/iomap.c
+++ b/fs/iomap.c
@@ -1463,6 +1463,28 @@ struct iomap_dio {
};
};
+int iomap_dio_iopoll(struct kiocb *kiocb, bool spin)
+{
+ struct request_queue *q = READ_ONCE(kiocb->private);
+
+ if (!q)
+ return 0;
+ return blk_poll(q, READ_ONCE(kiocb->ki_cookie), spin);
+}
+EXPORT_SYMBOL_GPL(iomap_dio_iopoll);
+
+static void iomap_dio_submit_bio(struct iomap_dio *dio, struct iomap *iomap,
+ struct bio *bio)
+{
+ atomic_inc(&dio->ref);
+
+ if (dio->iocb->ki_flags & IOCB_HIPRI)
+ bio_set_polled(bio, dio->iocb);
+
+ dio->submit.last_queue = bdev_get_queue(iomap->bdev);
+ dio->submit.cookie = submit_bio(bio);
+}
+
static ssize_t iomap_dio_complete(struct iomap_dio *dio)
{
struct kiocb *iocb = dio->iocb;
@@ -1575,7 +1597,7 @@ static void iomap_dio_bio_end_io(struct bio *bio)
}
}
-static blk_qc_t
+static void
iomap_dio_zero(struct iomap_dio *dio, struct iomap *iomap, loff_t pos,
unsigned len)
{
@@ -1589,15 +1611,10 @@ iomap_dio_zero(struct iomap_dio *dio, struct iomap *iomap, loff_t pos,
bio->bi_private = dio;
bio->bi_end_io = iomap_dio_bio_end_io;
- if (dio->iocb->ki_flags & IOCB_HIPRI)
- flags |= REQ_HIPRI;
-
get_page(page);
__bio_add_page(bio, page, len, 0);
bio_set_op_attrs(bio, REQ_OP_WRITE, flags);
-
- atomic_inc(&dio->ref);
- return submit_bio(bio);
+ iomap_dio_submit_bio(dio, iomap, bio);
}
static loff_t
@@ -1700,9 +1717,6 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
bio_set_pages_dirty(bio);
}
- if (dio->iocb->ki_flags & IOCB_HIPRI)
- bio->bi_opf |= REQ_HIPRI;
-
iov_iter_advance(dio->submit.iter, n);
dio->size += n;
@@ -1710,11 +1724,7 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
copied += n;
nr_pages = iov_iter_npages(&iter, BIO_MAX_PAGES);
-
- atomic_inc(&dio->ref);
-
- dio->submit.last_queue = bdev_get_queue(iomap->bdev);
- dio->submit.cookie = submit_bio(bio);
+ iomap_dio_submit_bio(dio, iomap, bio);
} while (nr_pages);
/*
@@ -1925,6 +1935,9 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
if (dio->flags & IOMAP_DIO_WRITE_FUA)
dio->flags &= ~IOMAP_DIO_NEED_SYNC;
+ WRITE_ONCE(iocb->ki_cookie, dio->submit.cookie);
+ WRITE_ONCE(iocb->private, dio->submit.last_queue);
+
/*
* We are about to drop our additional submission reference, which
* might be the last reference to the dio. There are three three
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index e47425071e65..60c2da41f0fc 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -1203,6 +1203,7 @@ const struct file_operations xfs_file_operations = {
.write_iter = xfs_file_write_iter,
.splice_read = generic_file_splice_read,
.splice_write = iter_file_splice_write,
+ .iopoll = iomap_dio_iopoll,
.unlocked_ioctl = xfs_file_ioctl,
#ifdef CONFIG_COMPAT
.compat_ioctl = xfs_file_compat_ioctl,
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 9a4258154b25..0fefb5455bda 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -162,6 +162,7 @@ typedef int (iomap_dio_end_io_t)(struct kiocb *iocb, ssize_t ret,
unsigned flags);
ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
const struct iomap_ops *ops, iomap_dio_end_io_t end_io);
+int iomap_dio_iopoll(struct kiocb *kiocb, bool spin);
#ifdef CONFIG_SWAP
struct file;
--
2.17.1
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply related
* [PATCH 03/18] block: add bio_set_polled() helper
From: Jens Axboe @ 2019-02-07 19:55 UTC (permalink / raw)
To: linux-aio, linux-block, linux-api
Cc: hch, jmoyer, avi, jannh, viro, Jens Axboe
In-Reply-To: <20190207195552.22770-1-axboe@kernel.dk>
For the upcoming async polled IO, we can't sleep allocating requests.
If we do, then we introduce a deadlock where the submitter already
has async polled IO in-flight, but can't wait for them to complete
since polled requests must be active found and reaped.
Utilize the helper in the blockdev DIRECT_IO code.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
fs/block_dev.c | 4 ++--
include/linux/bio.h | 14 ++++++++++++++
2 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/fs/block_dev.c b/fs/block_dev.c
index f18d076a2596..392e2bfb636f 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -247,7 +247,7 @@ __blkdev_direct_IO_simple(struct kiocb *iocb, struct iov_iter *iter,
task_io_account_write(ret);
}
if (iocb->ki_flags & IOCB_HIPRI)
- bio.bi_opf |= REQ_HIPRI;
+ bio_set_polled(&bio, iocb);
qc = submit_bio(&bio);
for (;;) {
@@ -415,7 +415,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages)
nr_pages = iov_iter_npages(iter, BIO_MAX_PAGES);
if (!nr_pages) {
if (iocb->ki_flags & IOCB_HIPRI)
- bio->bi_opf |= REQ_HIPRI;
+ bio_set_polled(bio, iocb);
qc = submit_bio(bio);
WRITE_ONCE(iocb->ki_cookie, qc);
diff --git a/include/linux/bio.h b/include/linux/bio.h
index 7380b094dcca..f6f0a2b3cbc8 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -823,5 +823,19 @@ static inline int bio_integrity_add_page(struct bio *bio, struct page *page,
#endif /* CONFIG_BLK_DEV_INTEGRITY */
+/*
+ * Mark a bio as polled. Note that for async polled IO, the caller must
+ * expect -EWOULDBLOCK if we cannot allocate a request (or other resources).
+ * We cannot block waiting for requests on polled IO, as those completions
+ * must be found by the caller. This is different than IRQ driven IO, where
+ * it's safe to wait for IO to complete.
+ */
+static inline void bio_set_polled(struct bio *bio, struct kiocb *kiocb)
+{
+ bio->bi_opf |= REQ_HIPRI;
+ if (!is_sync_kiocb(kiocb))
+ bio->bi_opf |= REQ_NOWAIT;
+}
+
#endif /* CONFIG_BLOCK */
#endif /* __LINUX_BIO_H */
--
2.17.1
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply related
* [PATCH 02/18] block: wire up block device iopoll method
From: Jens Axboe @ 2019-02-07 19:55 UTC (permalink / raw)
To: linux-aio, linux-block, linux-api
Cc: hch, jmoyer, avi, jannh, viro, Jens Axboe
In-Reply-To: <20190207195552.22770-1-axboe@kernel.dk>
From: Christoph Hellwig <hch@lst.de>
Just call blk_poll on the iocb cookie, we can derive the block device
from the inode trivially.
Reviewed-by: Johannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
fs/block_dev.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/fs/block_dev.c b/fs/block_dev.c
index 58a4c1217fa8..f18d076a2596 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -293,6 +293,14 @@ struct blkdev_dio {
static struct bio_set blkdev_dio_pool;
+static int blkdev_iopoll(struct kiocb *kiocb, bool wait)
+{
+ struct block_device *bdev = I_BDEV(kiocb->ki_filp->f_mapping->host);
+ struct request_queue *q = bdev_get_queue(bdev);
+
+ return blk_poll(q, READ_ONCE(kiocb->ki_cookie), wait);
+}
+
static void blkdev_bio_end_io(struct bio *bio)
{
struct blkdev_dio *dio = bio->bi_private;
@@ -410,6 +418,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages)
bio->bi_opf |= REQ_HIPRI;
qc = submit_bio(bio);
+ WRITE_ONCE(iocb->ki_cookie, qc);
break;
}
@@ -2076,6 +2085,7 @@ const struct file_operations def_blk_fops = {
.llseek = block_llseek,
.read_iter = blkdev_read_iter,
.write_iter = blkdev_write_iter,
+ .iopoll = blkdev_iopoll,
.mmap = generic_file_mmap,
.fsync = blkdev_fsync,
.unlocked_ioctl = block_ioctl,
--
2.17.1
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply related
* [PATCH 01/18] fs: add an iopoll method to struct file_operations
From: Jens Axboe @ 2019-02-07 19:55 UTC (permalink / raw)
To: linux-aio, linux-block, linux-api
Cc: hch, jmoyer, avi, jannh, viro, Jens Axboe
In-Reply-To: <20190207195552.22770-1-axboe@kernel.dk>
From: Christoph Hellwig <hch@lst.de>
This new methods is used to explicitly poll for I/O completion for an
iocb. It must be called for any iocb submitted asynchronously (that
is with a non-null ki_complete) which has the IOCB_HIPRI flag set.
The method is assisted by a new ki_cookie field in struct iocb to store
the polling cookie.
Reviewed-by: Johannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
Documentation/filesystems/vfs.txt | 3 +++
include/linux/fs.h | 2 ++
2 files changed, 5 insertions(+)
diff --git a/Documentation/filesystems/vfs.txt b/Documentation/filesystems/vfs.txt
index 8dc8e9c2913f..761c6fd24a53 100644
--- a/Documentation/filesystems/vfs.txt
+++ b/Documentation/filesystems/vfs.txt
@@ -857,6 +857,7 @@ struct file_operations {
ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);
ssize_t (*read_iter) (struct kiocb *, struct iov_iter *);
ssize_t (*write_iter) (struct kiocb *, struct iov_iter *);
+ int (*iopoll)(struct kiocb *kiocb, bool spin);
int (*iterate) (struct file *, struct dir_context *);
int (*iterate_shared) (struct file *, struct dir_context *);
__poll_t (*poll) (struct file *, struct poll_table_struct *);
@@ -902,6 +903,8 @@ otherwise noted.
write_iter: possibly asynchronous write with iov_iter as source
+ iopoll: called when aio wants to poll for completions on HIPRI iocbs
+
iterate: called when the VFS needs to read the directory contents
iterate_shared: called when the VFS needs to read the directory contents
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 29d8e2cfed0e..dedcc2e9265c 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -310,6 +310,7 @@ struct kiocb {
int ki_flags;
u16 ki_hint;
u16 ki_ioprio; /* See linux/ioprio.h */
+ unsigned int ki_cookie; /* for ->iopoll */
} __randomize_layout;
static inline bool is_sync_kiocb(struct kiocb *kiocb)
@@ -1787,6 +1788,7 @@ struct file_operations {
ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);
ssize_t (*read_iter) (struct kiocb *, struct iov_iter *);
ssize_t (*write_iter) (struct kiocb *, struct iov_iter *);
+ int (*iopoll)(struct kiocb *kiocb, bool spin);
int (*iterate) (struct file *, struct dir_context *);
int (*iterate_shared) (struct file *, struct dir_context *);
__poll_t (*poll) (struct file *, struct poll_table_struct *);
--
2.17.1
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply related
* [PATCHSET v12] io_uring IO interface
From: Jens Axboe @ 2019-02-07 19:55 UTC (permalink / raw)
To: linux-aio, linux-block, linux-api; +Cc: hch, jmoyer, avi, jannh, viro
Here's v12 of the io_uring project. This is the Al Viro special, where
Al tries to beat into my head how UNIX fd passing will mess you up. I
think we have all cases handled now. I've added the resulting test case
into the liburing test/ directory.
Outside of that, various little cleanups and fixes, and a revert of
using FOLL_ANON to map IO buffers. This makes it fail with huge pages,
something that we can (and do want to) support. We're now back to just
checking for a file backing in the vma.
As far as I'm concerned, this project is ready to get staged for 5.1.
Please do review carefully so we can fix any minor nits that might still
exist.
The liburing git repo has a full set of man pages for this, though they
could probably still use a bit of polish. I'd also like to see a
io_uring(7) man page to describe the overall design of the project,
expect that in the not-so-distant future. You can clone that here:
git://git.kernel.dk/liburing
Patches are against 5.0-rc5, and can also be found in my io_uring branch
here:
git://git.kernel.dk/linux-block io_uring
Since v11:
- Get rid of the need for storing a files_struct
- Protect against release loop when UNIX fd passing is used
- Retain kiocb state for EAGAIN async retry
- Cleanup io memory accounting
- Revert to using non-FOLL_ANON, as we do want to support huge pages
- Fix uid leak
- Address various review comments
- Rebase on v5.0-rc5
Documentation/filesystems/vfs.txt | 3 +
arch/x86/entry/syscalls/syscall_32.tbl | 3 +
arch/x86/entry/syscalls/syscall_64.tbl | 3 +
block/bio.c | 59 +-
fs/Makefile | 1 +
fs/block_dev.c | 19 +-
fs/file.c | 15 +-
fs/file_table.c | 9 +-
fs/gfs2/file.c | 2 +
fs/io_uring.c | 2741 ++++++++++++++++++++++++
fs/iomap.c | 48 +-
fs/xfs/xfs_file.c | 1 +
include/linux/bio.h | 14 +
include/linux/blk_types.h | 1 +
include/linux/file.h | 2 +
include/linux/fs.h | 15 +-
include/linux/iomap.h | 1 +
include/linux/sched/user.h | 2 +-
include/linux/syscalls.h | 8 +
include/net/af_unix.h | 1 +
include/uapi/asm-generic/unistd.h | 8 +-
include/uapi/linux/io_uring.h | 142 ++
init/Kconfig | 9 +
kernel/sys_ni.c | 3 +
net/unix/af_unix.c | 2 +-
net/unix/garbage.c | 3 +
26 files changed, 3073 insertions(+), 42 deletions(-)
--
Jens Axboe
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Miklos Szeredi @ 2019-02-07 19:08 UTC (permalink / raw)
To: Al Viro
Cc: Jens Axboe, Jann Horn, linux-aio, linux-block, Linux API,
Christoph Hellwig, Jeff Moyer, Avi Kivity, linux-fsdevel
In-Reply-To: <20190207162605.GD2217@ZenIV.linux.org.uk>
On Thu, Feb 7, 2019 at 5:26 PM Al Viro <viro@zeniv.linux.org.uk> wrote:
> I'm trying to put together some formal description of what's going on in there.
> Another question, BTW: updates of user->unix_inflight would seem to be movable
> into the callers of unix_{not,}inflight(). Any objections against lifting
> it into unix_{attach,detach}_fds()? We do, after all, have fp->count right
> there, so what's the point incrementing/decrementing the sucker one-by-one?
> _And_ we are checking it right there (in too_many_unix_fds() called from
> unix_attach_fds())...
I see no issues with that.
Also shouldn't the rlimit check be made against user->unix_inflight +
fp->count? Althought I'm not quite following if fp->user can end up
different from current_user() and what should happen in that case...
Thanks,
Miklos
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Jens Axboe @ 2019-02-07 18:58 UTC (permalink / raw)
To: Al Viro
Cc: Jann Horn, linux-aio, linux-block, Linux API, hch, jmoyer, avi,
linux-fsdevel
In-Reply-To: <73e23146-2138-5a46-46ed-9c7f1f912a04@kernel.dk>
On 2/7/19 11:45 AM, Jens Axboe wrote:
> On 2/6/19 9:00 PM, Al Viro wrote:
>> On Wed, Feb 06, 2019 at 06:41:00AM -0700, Jens Axboe wrote:
>>> On 2/5/19 5:56 PM, Al Viro wrote:
>>>> On Tue, Feb 05, 2019 at 12:08:25PM -0700, Jens Axboe wrote:
>>>>> Proof is in the pudding, here's the main commit introducing io_uring
>>>>> and now wiring it up to the AF_UNIX garbage collection:
>>>>>
>>>>> http://git.kernel.dk/cgit/linux-block/commit/?h=io_uring&id=158e6f42b67d0abe9ee84886b96ca8c4b3d3dfd5
>>>>>
>>>>> How does that look?
>>>>
>>>> In a word - wrong. Some theory: garbage collector assumes that there is
>>>> a subset of file references such that
>>>> * for all files with such references there's an associated unix_sock.
>>>> * all such references are stored in SCM_RIGHTS datagrams that can be
>>>> found by the garbage collector (currently: for data-bearing AF_UNIX sockets -
>>>> queued SCM_RIGHTS datagrams, for listeners - SCM_RIGHTS datagrams sent via
>>>> yet-to-be-accepted connections).
>>>> * there is an efficient way to count those references for given file
>>>> (->inflight of the corresponding unix_sock).
>>>> * removal of those references would render the graph acyclic.
>>>> * file can _NOT_ be subject to syscalls unless there are references
>>>> to it outside of that subset.
>>>
>>> IOW, we cannot use fget() for registering files, and we still need fget/fput
>>> in the fast path to retain safe use of the file. If I'm understanding you
>>> correctly?
>>
>> No. *ALL* references (inflight and not) are the same for file->f_count.
>> unix_inflight() does not grab a new reference to file; it only says that
>> reference passed to it by the caller is now an in-flight one.
>>
>> OK, braindump time:
>
> [snip]
>
> This is great info, and I think it belongs in Documentation/ somewhere.
> Not sure I've ever seen such a good and detailed dump of this before.
>
>> What you are about to add is *ANOTHER* kind of loops - references
>> to files in the "registered" set are pinned down by owning io_uring.
>>
>> That would invalidate just about every assumption made the garbage
>> collector - even if you forbid to register io_uring itself, you
>> still can register both ends of AF_UNIX socket pair, then pass
>> io_uring in SCM_RIGHTS over that, then close all descriptors involved.
>> From the garbage collector point of view all sockets have external
>> references, so there's nothing to collect. In fact those external
>> references are only reachable if you have a reachable reference
>> to io_uring, so we get a leak.
>>
>> To make it work:
>> * have unix_sock created for each io_uring (as your code does)
>> * do *NOT* have unix_inflight() done at that point - it's
>> completely wrong there.
>> * file set registration becomes
>> * create and populate SCM_RIGHTS, with the same
>> fget()+grab an extra reference + unix_inflight() sequence.
>> Don't forget to have skb->destructor set to unix_destruct_scm
>> or equivalent thereof.
>> * remember UNIXCB(skb).fp - that'll give you your
>> array of struct file *, to use in lookups.
>> * queue it into your unix_sock
>> * do _one_ fput() for everything you've grabbed,
>> dropping one of two references you've taken.
>> * unregistering is simply skb_dequeue() + kfree_skb().
>> * in ->release() you do sock_release(); it'll do
>> everything you need (including unregistering the set, etc.)
>
> This is genius! I implemented this and it works. I've verified that the
> previous test app failed to release due to the loop, and with this in
> place, once the GC kicks in, the io_uring is released appropriately.
>
>> The hairiest part is the file set registration, of course -
>> there's almost certainly a helper or two buried in that thing;
>> simply exposing all the required net/unix/af_unix.c bits is
>> ucking fugly.
>
> Outside of the modification to unix_get_socket(), the only change I had
> to make was to ensure that unix_destruct_scm() is available to io_uring.
> No other changes needed.
>
>> I'm not sure what you propose for non-registered descriptors -
>> _any_ struct file reference that outlives the return from syscall
>> stored in some io_uring-attached data structure is has exact same
>> loop (and leak) problem. And if you mean to have it dropped before
>> return from syscall, I'm afraid I don't follow you. How would
>> that be done?
>>
>> Again, "io_uring descriptor can't be used in those requests" does
>> not help at all - use a socket instead, pass the io_uring fd over
>> it in SCM_RIGHTS and you are back to square 1.
>
> I wasn't proposing to fput() before return, otherwise I can't hang on to
> that file *.
>
> Right now for async punt, we don't release the reference, and then we
> fput() when IO completes. According to what you're saying here, that's
> not good enough. Correct me if I'm wrong, but what if we:
>
> 1) For non-sock/io_uring fds, the current approach is sufficient
> 2) Disallow io_uring fd, doesn't make sense anyway
>
> That leaves the socket fd, which is problematic. Should be solvable by
> allocating an skb and marking that file inflight?
Actually, we can just NOT set NOWAIT for types we don't support. That
means we'll never punt to async context for those.
--
Jens Axboe
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Jens Axboe @ 2019-02-07 18:45 UTC (permalink / raw)
To: Al Viro
Cc: Jann Horn, linux-aio, linux-block, Linux API, hch, jmoyer, avi,
linux-fsdevel
In-Reply-To: <20190207040058.GW2217@ZenIV.linux.org.uk>
On 2/6/19 9:00 PM, Al Viro wrote:
> On Wed, Feb 06, 2019 at 06:41:00AM -0700, Jens Axboe wrote:
>> On 2/5/19 5:56 PM, Al Viro wrote:
>>> On Tue, Feb 05, 2019 at 12:08:25PM -0700, Jens Axboe wrote:
>>>> Proof is in the pudding, here's the main commit introducing io_uring
>>>> and now wiring it up to the AF_UNIX garbage collection:
>>>>
>>>> http://git.kernel.dk/cgit/linux-block/commit/?h=io_uring&id=158e6f42b67d0abe9ee84886b96ca8c4b3d3dfd5
>>>>
>>>> How does that look?
>>>
>>> In a word - wrong. Some theory: garbage collector assumes that there is
>>> a subset of file references such that
>>> * for all files with such references there's an associated unix_sock.
>>> * all such references are stored in SCM_RIGHTS datagrams that can be
>>> found by the garbage collector (currently: for data-bearing AF_UNIX sockets -
>>> queued SCM_RIGHTS datagrams, for listeners - SCM_RIGHTS datagrams sent via
>>> yet-to-be-accepted connections).
>>> * there is an efficient way to count those references for given file
>>> (->inflight of the corresponding unix_sock).
>>> * removal of those references would render the graph acyclic.
>>> * file can _NOT_ be subject to syscalls unless there are references
>>> to it outside of that subset.
>>
>> IOW, we cannot use fget() for registering files, and we still need fget/fput
>> in the fast path to retain safe use of the file. If I'm understanding you
>> correctly?
>
> No. *ALL* references (inflight and not) are the same for file->f_count.
> unix_inflight() does not grab a new reference to file; it only says that
> reference passed to it by the caller is now an in-flight one.
>
> OK, braindump time:
[snip]
This is great info, and I think it belongs in Documentation/ somewhere.
Not sure I've ever seen such a good and detailed dump of this before.
> What you are about to add is *ANOTHER* kind of loops - references
> to files in the "registered" set are pinned down by owning io_uring.
>
> That would invalidate just about every assumption made the garbage
> collector - even if you forbid to register io_uring itself, you
> still can register both ends of AF_UNIX socket pair, then pass
> io_uring in SCM_RIGHTS over that, then close all descriptors involved.
> From the garbage collector point of view all sockets have external
> references, so there's nothing to collect. In fact those external
> references are only reachable if you have a reachable reference
> to io_uring, so we get a leak.
>
> To make it work:
> * have unix_sock created for each io_uring (as your code does)
> * do *NOT* have unix_inflight() done at that point - it's
> completely wrong there.
> * file set registration becomes
> * create and populate SCM_RIGHTS, with the same
> fget()+grab an extra reference + unix_inflight() sequence.
> Don't forget to have skb->destructor set to unix_destruct_scm
> or equivalent thereof.
> * remember UNIXCB(skb).fp - that'll give you your
> array of struct file *, to use in lookups.
> * queue it into your unix_sock
> * do _one_ fput() for everything you've grabbed,
> dropping one of two references you've taken.
> * unregistering is simply skb_dequeue() + kfree_skb().
> * in ->release() you do sock_release(); it'll do
> everything you need (including unregistering the set, etc.)
This is genius! I implemented this and it works. I've verified that the
previous test app failed to release due to the loop, and with this in
place, once the GC kicks in, the io_uring is released appropriately.
> The hairiest part is the file set registration, of course -
> there's almost certainly a helper or two buried in that thing;
> simply exposing all the required net/unix/af_unix.c bits is
> ucking fugly.
Outside of the modification to unix_get_socket(), the only change I had
to make was to ensure that unix_destruct_scm() is available to io_uring.
No other changes needed.
> I'm not sure what you propose for non-registered descriptors -
> _any_ struct file reference that outlives the return from syscall
> stored in some io_uring-attached data structure is has exact same
> loop (and leak) problem. And if you mean to have it dropped before
> return from syscall, I'm afraid I don't follow you. How would
> that be done?
>
> Again, "io_uring descriptor can't be used in those requests" does
> not help at all - use a socket instead, pass the io_uring fd over
> it in SCM_RIGHTS and you are back to square 1.
I wasn't proposing to fput() before return, otherwise I can't hang on to
that file *.
Right now for async punt, we don't release the reference, and then we
fput() when IO completes. According to what you're saying here, that's
not good enough. Correct me if I'm wrong, but what if we:
1) For non-sock/io_uring fds, the current approach is sufficient
2) Disallow io_uring fd, doesn't make sense anyway
That leaves the socket fd, which is problematic. Should be solvable by
allocating an skb and marking that file inflight?
--
Jens Axboe
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Al Viro @ 2019-02-07 16:51 UTC (permalink / raw)
To: Jens Axboe
Cc: Jann Horn, linux-aio, linux-block, Linux API, hch, jmoyer, avi,
linux-fsdevel
In-Reply-To: <20190207163047.GE2217@ZenIV.linux.org.uk>
On Thu, Feb 07, 2019 at 04:30:47PM +0000, Al Viro wrote:
> Well, yes - once you receive it, you obviously have no references
> sitting in SCM_RIGHTS anymore.
>
> Get rid of recv_fd() there (along with fork(), while we are at it - what's
> it for?) and just do send_fd + these 3 close (or just exit, for that matter).
If you pardon a bad ASCII graphics,
ring_fd sv[0] sv[1] (descriptors)
| | |
V V V
[io_uring] [socket1] [socket2] (struct file)
^ | ^ | ^ |
| | | | | |
| \_______________/__|________________/ | (after registering)
| | |
| V V
| [unix_sock1]<---->[unix_sock2] (struct unix_sock)
| |
| V
\----------------------------------[SCM_RIGHTS] (queue contents)
References from io_uring to other two struct file are added when you
register these suckers. Reference from SCM_RIGHTS appears when you
do send_fd(). Now, each file has two references to it. And
if you close all 3 descriptors (either explicitly, or by exiting)
you will be left with this graph:
[io_uring]------------\-------------------\
^ | |
| V V
| [socket1] [socket2]
| | |
| V V
| [unix_sock1]<---->[unix_sock2]
| |
| V
\----------------------------------[SCM_RIGHTS]
All struct file still have references, so they are all still alive,
->release() isn't called on any of them. And the entire thing
is obviously unreachable from the rest of data structures.
Of course recvmsg() would've removed the loop. The point is, with
that situation you *can't* get it called - you'd need to reach
socket2 to do that and you can't do that anymore.
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Jens Axboe @ 2019-02-07 16:35 UTC (permalink / raw)
To: Al Viro
Cc: Jann Horn, linux-aio, linux-block, Linux API, hch, jmoyer, avi,
linux-fsdevel
In-Reply-To: <20190207163047.GE2217@ZenIV.linux.org.uk>
On 2/7/19 9:30 AM, Al Viro wrote:
> On Thu, Feb 07, 2019 at 09:14:41AM -0700, Jens Axboe wrote:
>
>> I created a small app to do just that, and ran it and verified that
>> ->release() is called and the io_uring is released as expected. This
>> is run on the current -git branch, which has a socket backing for
>> the io_uring fd itself, but not for the registered files.
>>
>> What am I missing here? Attaching the program as a reference.
>
>> int main(int argc, char *argv[])
>> {
>> int sp[2], pid, ring_fd, ret;
>>
>> if (socketpair(AF_UNIX, SOCK_DGRAM, 0, sp) != 0) {
>> perror("Failed to create Unix-domain socket pair\n");
>> return 1;
>> }
>>
>> ring_fd = get_ring_fd();
>> if (ring_fd < 0)
>> return 1;
>>
>> ret = io_uring_register_files(ring_fd, sp[0], sp[1]);
>> if (ret < 0) {
>> perror("register files");
>> return 1;
>> }
>>
>> pid = fork();
>> if (pid) {
>> printf("Sending fd %d\n", ring_fd);
>>
>> send_fd(sp[0], ring_fd);
>> } else {
>> int fd;
>>
>> fd = recv_fd(sp[1]);
>
> Well, yes - once you receive it, you obviously have no references
> sitting in SCM_RIGHTS anymore.
>
> Get rid of recv_fd() there (along with fork(), while we are at it - what's
> it for?) and just do send_fd + these 3 close (or just exit, for that matter).
Ah got it, yes you are right, that does leak.
Thanks for the other (very) detailed note, I'll add a socket backing for the
registered files. I'll respond to the other details in there a bit later.
--
Jens Axboe
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Al Viro @ 2019-02-07 16:30 UTC (permalink / raw)
To: Jens Axboe
Cc: Jann Horn, linux-aio, linux-block, Linux API, hch, jmoyer, avi,
linux-fsdevel
In-Reply-To: <51f167c6-aacc-dd5b-4481-7168c6f8b413@kernel.dk>
On Thu, Feb 07, 2019 at 09:14:41AM -0700, Jens Axboe wrote:
> I created a small app to do just that, and ran it and verified that
> ->release() is called and the io_uring is released as expected. This
> is run on the current -git branch, which has a socket backing for
> the io_uring fd itself, but not for the registered files.
>
> What am I missing here? Attaching the program as a reference.
> int main(int argc, char *argv[])
> {
> int sp[2], pid, ring_fd, ret;
>
> if (socketpair(AF_UNIX, SOCK_DGRAM, 0, sp) != 0) {
> perror("Failed to create Unix-domain socket pair\n");
> return 1;
> }
>
> ring_fd = get_ring_fd();
> if (ring_fd < 0)
> return 1;
>
> ret = io_uring_register_files(ring_fd, sp[0], sp[1]);
> if (ret < 0) {
> perror("register files");
> return 1;
> }
>
> pid = fork();
> if (pid) {
> printf("Sending fd %d\n", ring_fd);
>
> send_fd(sp[0], ring_fd);
> } else {
> int fd;
>
> fd = recv_fd(sp[1]);
Well, yes - once you receive it, you obviously have no references
sitting in SCM_RIGHTS anymore.
Get rid of recv_fd() there (along with fork(), while we are at it - what's
it for?) and just do send_fd + these 3 close (or just exit, for that matter).
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Al Viro @ 2019-02-07 16:26 UTC (permalink / raw)
To: Miklos Szeredi
Cc: Jens Axboe, Jann Horn, linux-aio, linux-block, Linux API,
Christoph Hellwig, Jeff Moyer, Avi Kivity, linux-fsdevel
In-Reply-To: <CAJfpegvayM_ZmuVetfYzpK+qqZJPkVzfQVRQ7OO98f1=reqiYA@mail.gmail.com>
On Thu, Feb 07, 2019 at 04:27:13PM +0100, Miklos Szeredi wrote:
> On Thu, Feb 7, 2019 at 4:20 PM Al Viro <viro@zeniv.linux.org.uk> wrote:
> >
> > On Thu, Feb 07, 2019 at 03:20:06PM +0100, Miklos Szeredi wrote:
> >
> > > > Am I right assuming that this queue-modifying operation is accept(), removing
> > > > an embryo unix_sock from the queue of listener and thus hiding SCM_RIGHTS in
> > > > _its_ queue from scan_children()?
> > >
> > > Hmm... How about just receiving an SCM_RIGHTS socket (which was a
> > > candidate) from the queue of the peeked socket?
> >
> > Right, skb unlinked before unix_detach_fds(). I was actually thinking of a stream
> > case, where unlink is done after that...
> >
> > *grumble*
> >
> > The entire thing is far too brittle for my taste ;-/
>
> If it gets used as part of io_uring, I guess it's worth a fresh look.
> I wrote it without basically any experience with either networking or
> garbage collecting, so no wonder it has rough edges.
It had a plenty of those edges before your changes as well - I'm not blaming you
for that mess, in case that's not obvious from what I'd written.
I'm trying to put together some formal description of what's going on in there.
Another question, BTW: updates of user->unix_inflight would seem to be movable
into the callers of unix_{not,}inflight(). Any objections against lifting
it into unix_{attach,detach}_fds()? We do, after all, have fp->count right
there, so what's the point incrementing/decrementing the sucker one-by-one?
_And_ we are checking it right there (in too_many_unix_fds() called from
unix_attach_fds())...
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Jens Axboe @ 2019-02-07 16:14 UTC (permalink / raw)
To: Al Viro
Cc: Jann Horn, linux-aio, linux-block, Linux API, hch, jmoyer, avi,
linux-fsdevel
In-Reply-To: <20190207040547.GX2217@ZenIV.linux.org.uk>
[-- Attachment #1: Type: text/plain, Size: 1524 bytes --]
On 2/6/19 9:05 PM, Al Viro wrote:
> On Wed, Feb 06, 2019 at 10:56:41AM -0700, Jens Axboe wrote:
>> On 2/5/19 6:01 PM, Al Viro wrote:
>>> On Tue, Feb 05, 2019 at 05:27:29PM -0700, Jens Axboe wrote:
>>>
>>>> This should be better, passes some basic testing, too:
>>>>
>>>> http://git.kernel.dk/cgit/linux-block/commit/?h=io_uring&id=01a93aa784319a02ccfa6523371b93401c9e0073
>>>>
>>>> Verified that we're grabbing the right refs, and don't hold any
>>>> ourselves. For the file registration, forbid registration of the
>>>> io_uring fd, as that is pointless and will introduce a loop regardless
>>>> of fd passing.
>>>
>>> *shrug*
>>>
>>> So pass it to AF_UNIX socket and register _that_ - does't change the
>>> underlying problem.
>>
>> Maybe I'm being dense here, but it's an f_op match. Should catch a
>> passed fd as well, correct?
>
> f_op match on _what_?
>
>> With that, how can there be a loop?
>
> io_uring_fd = ....
> socketpair(PF_UNIX, SOCK_STREAM, 0, sock_fds);
> register sock_fds[0] and sock_fds[1] to io_uring_fd
> send SCM_RIGHTS datagram with io_uring_fd to sock_fds[0]
> close sock_fds[0], sock_fds[1] and io_uring_fd
>
> And there's your unreachable loop.
I created a small app to do just that, and ran it and verified that
->release() is called and the io_uring is released as expected. This
is run on the current -git branch, which has a socket backing for
the io_uring fd itself, but not for the registered files.
What am I missing here? Attaching the program as a reference.
--
Jens Axboe
[-- Attachment #2: viro.c --]
[-- Type: text/x-csrc, Size: 2969 bytes --]
#include <stdio.h>
#include <stdlib.h>
#include <stddef.h>
#include <signal.h>
#include <inttypes.h>
#include <sys/types.h>
#include <sys/syscall.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
#include <linux/fs.h>
struct io_sqring_offsets {
__u32 head;
__u32 tail;
__u32 ring_mask;
__u32 ring_entries;
__u32 flags;
__u32 dropped;
__u32 array;
__u32 resv[3];
};
struct io_cqring_offsets {
__u32 head;
__u32 tail;
__u32 ring_mask;
__u32 ring_entries;
__u32 overflow;
__u32 cqes;
__u32 resv[4];
};
struct io_uring_params {
__u32 sq_entries;
__u32 cq_entries;
__u32 flags;
__u32 sq_thread_cpu;
__u32 sq_thread_idle;
__u32 resv[5];
struct io_sqring_offsets sq_off;
struct io_cqring_offsets cq_off;
};
#define IORING_REGISTER_FILES 2
#define __NR_sys_io_uring_setup 425
#define __NR_sys_io_uring_register 427
static int io_uring_register_files(int ring_fd, int fd1, int fd2)
{
__s32 *fds;
fds = calloc(2, sizeof(__s32));
fds[0] = fd1;
fds[1] = fd2;
return syscall(__NR_sys_io_uring_register, ring_fd,
IORING_REGISTER_FILES, fds, 2);
}
static int io_uring_setup(unsigned entries, struct io_uring_params *p)
{
return syscall(__NR_sys_io_uring_setup, entries, p);
}
static int get_ring_fd(void)
{
struct io_uring_params p;
int fd;
memset(&p, 0, sizeof(p));
fd = io_uring_setup(2, &p);
if (fd < 0) {
perror("io_uring_setup");
return -1;
}
return fd;
}
static void send_fd(int socket, int fd)
{
char buf[CMSG_SPACE(sizeof(fd))];
struct cmsghdr *cmsg;
struct msghdr msg;
memset(buf, 0, sizeof(buf));
memset(&msg, 0, sizeof(msg));
msg.msg_control = buf;
msg.msg_controllen = sizeof(buf);
cmsg = CMSG_FIRSTHDR(&msg);
cmsg->cmsg_level = SOL_SOCKET;
cmsg->cmsg_type = SCM_RIGHTS;
cmsg->cmsg_len = CMSG_LEN(sizeof(fd));
memmove(CMSG_DATA(cmsg), &fd, sizeof(fd));
msg.msg_controllen = CMSG_SPACE(sizeof(fd));
if (sendmsg(socket, &msg, 0) < 0)
perror("sendmsg");
}
static int recv_fd(int socket)
{
struct msghdr msg;
char c_buffer[256];
struct cmsghdr *cmsg;
int fd;
memset(&msg, 0, sizeof(msg));
msg.msg_control = c_buffer;
msg.msg_controllen = sizeof(c_buffer);
if (recvmsg(socket, &msg, 0) < 0)
perror("recvmsg\n");
cmsg = CMSG_FIRSTHDR(&msg);
memmove(&fd, CMSG_DATA(cmsg), sizeof(fd));
return fd;
}
int main(int argc, char *argv[])
{
int sp[2], pid, ring_fd, ret;
if (socketpair(AF_UNIX, SOCK_DGRAM, 0, sp) != 0) {
perror("Failed to create Unix-domain socket pair\n");
return 1;
}
ring_fd = get_ring_fd();
if (ring_fd < 0)
return 1;
ret = io_uring_register_files(ring_fd, sp[0], sp[1]);
if (ret < 0) {
perror("register files");
return 1;
}
pid = fork();
if (pid) {
printf("Sending fd %d\n", ring_fd);
send_fd(sp[0], ring_fd);
} else {
int fd;
fd = recv_fd(sp[1]);
printf("Got fd %d\n", fd);
close(fd);
}
usleep(500000);
close(ring_fd);
close(sp[0]);
close(sp[1]);
return 0;
}
^ permalink raw reply
* Re: [PATCH 16/32] x86/vdso: Generate vdso{,32}-timens.lds
From: Dmitry Safonov @ 2019-02-07 16:11 UTC (permalink / raw)
To: Rasmus Villemoes, Dmitry Safonov, linux-kernel
Cc: Adrian Reber, Andrei Vagin, Andrei Vagin, Andy Lutomirski,
Andy Tucker, Arnd Bergmann, Christian Brauner, Cyrill Gorcunov,
Eric W. Biederman, H. Peter Anvin, Ingo Molnar, Jeff Dike,
Oleg Nesterov, Pavel Emelyanov, Shuah Khan, Thomas Gleixner,
containers, criu, linux-api, x86
In-Reply-To: <a2bbe610-7553-0ba6-b58b-1013e034b8f6@rasmusvillemoes.dk>
Hi Rasmus,
On 2/7/19 8:31 AM, Rasmus Villemoes wrote:
> These (14-19, if I'm reading them right) seems to add quite a lot of
> complexity and fragility to the build, and other architectures would
> probably have to add something similar to their vdso builds.
>
> I'm wondering why not make the rule be that a timens takes effect on
> next execve?
I believe, it would make setns() syscall much tricker than wanted:
At this moment the only exception is pidns which changes ns of the child
and not the process-callee.
If exec() would be required to join timens - it may be a challenging
problem for container systems: in order to enter it one needs to
exec("/proc/self/exe") and add some new arguments/options.
Furthermore, it seems to me that to enter container with this semantics,
one needs to enter timens before entering mountns.
IOW, I believe, this would move complexity from kernel build time to
userspace ABI. And I guess, it would require much more logic to
re-create possibly nested namespaces hierarchy.
Rather I've considered using some kind of dynamic patching on vdso_init():
o static_branch - it would add some nops to !timens vdso
o something new like static_retpoline which would put RET over call to
clk_to_ns(), shouldn't be a rocket since.
But in my point of view, if something can be done in compile time
instead of patching code dynamically - than it reduces the complexity
(lesser depends on what compiler/toolchain does).
Thanks,
Dmitry
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Miklos Szeredi @ 2019-02-07 15:27 UTC (permalink / raw)
To: Al Viro
Cc: Jens Axboe, Jann Horn, linux-aio, linux-block, Linux API,
Christoph Hellwig, Jeff Moyer, Avi Kivity, linux-fsdevel
In-Reply-To: <20190207152002.GC2217@ZenIV.linux.org.uk>
On Thu, Feb 7, 2019 at 4:20 PM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> On Thu, Feb 07, 2019 at 03:20:06PM +0100, Miklos Szeredi wrote:
>
> > > Am I right assuming that this queue-modifying operation is accept(), removing
> > > an embryo unix_sock from the queue of listener and thus hiding SCM_RIGHTS in
> > > _its_ queue from scan_children()?
> >
> > Hmm... How about just receiving an SCM_RIGHTS socket (which was a
> > candidate) from the queue of the peeked socket?
>
> Right, skb unlinked before unix_detach_fds(). I was actually thinking of a stream
> case, where unlink is done after that...
>
> *grumble*
>
> The entire thing is far too brittle for my taste ;-/
If it gets used as part of io_uring, I guess it's worth a fresh look.
I wrote it without basically any experience with either networking or
garbage collecting, so no wonder it has rough edges.
Thanks,
Miklos
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Al Viro @ 2019-02-07 15:20 UTC (permalink / raw)
To: Miklos Szeredi
Cc: Jens Axboe, Jann Horn, linux-aio, linux-block, Linux API,
Christoph Hellwig, Jeff Moyer, Avi Kivity, linux-fsdevel
In-Reply-To: <CAJfpegutRaXZ8EC_Am_PomxwdmSoH7eQkxAewaLfUrMALjym-w@mail.gmail.com>
On Thu, Feb 07, 2019 at 03:20:06PM +0100, Miklos Szeredi wrote:
> > Am I right assuming that this queue-modifying operation is accept(), removing
> > an embryo unix_sock from the queue of listener and thus hiding SCM_RIGHTS in
> > _its_ queue from scan_children()?
>
> Hmm... How about just receiving an SCM_RIGHTS socket (which was a
> candidate) from the queue of the peeked socket?
Right, skb unlinked before unix_detach_fds(). I was actually thinking of a stream
case, where unlink is done after that...
*grumble*
The entire thing is far too brittle for my taste ;-/
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Miklos Szeredi @ 2019-02-07 14:20 UTC (permalink / raw)
To: Al Viro
Cc: Jens Axboe, Jann Horn, linux-aio, linux-block, Linux API,
Christoph Hellwig, Jeff Moyer, Avi Kivity, linux-fsdevel
In-Reply-To: <20190207133135.GZ2217@ZenIV.linux.org.uk>
On Thu, Feb 7, 2019 at 2:31 PM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> On Thu, Feb 07, 2019 at 10:22:53AM +0100, Miklos Szeredi wrote:
> > On Thu, Feb 07, 2019 at 04:00:59AM +0000, Al Viro wrote:
> >
> > > So in theory it would be possible to have
> > > * thread A: sendmsg() has SCM_RIGHTS created and populated,
> > > complete with file refcount and ->inflight increments implied,
> > > at which point it gets preempted and loses the timeslice.
> > > * thread B: gets to run and removes all references
> > > from descriptor table it shares with thread A.
> > > * on another CPU we have garbage collector triggered;
> > > it determines the set of potentially unreachable unix_sock and
> > > everything in our SCM_RIGHTS _is_ in that set, now that no
> > > other references remain.
> > > * on the first CPU, thread A regains the timeslice
> > > and inserts its SCM_RIGHTS into queue. And it does contain
> > > references to sockets from the candidate set of running
> > > garbage collector, confusing the hell out of it.
> >
> > Reminds me: long time ago there was a bug report, and based on that I found a
> > bug in MSG_PEEK handling (not confirmed to have fixed the reported bug). This
> > fix, although pretty simple, got lost somehow. While unix gc code is in your
> > head, can you please review and I'll resend through davem?
>
> Umm... I think the bug is real (something that looks like an eviction
> candidate, but actually is referenced from the reachable queue might
> get peeked via that queue, then have _its_ queue modified via new
> external reference, all between two passes over that queue, confusing the
> fuck out of unix_gc()), but I think the fix is an overkill...
>
> Am I right assuming that this queue-modifying operation is accept(), removing
> an embryo unix_sock from the queue of listener and thus hiding SCM_RIGHTS in
> _its_ queue from scan_children()?
Hmm... How about just receiving an SCM_RIGHTS socket (which was a
candidate) from the queue of the peeked socket?
> Let me think of it a bit, OK? While we are at it, some questions from digging
> through the current net/unix/garbage.c:
> 1) is there any need for ->inflight to be atomic? All accesses are under
> unix_gc_lock, after all...
Seems so. Probably historic.
> 2) pumping unix_gc on each sodding reference in SCM_RIGHTS (within
> unix_notinflight()/unix_inflight()) looks atrocious... wouldn't it be better to
> hold it over that loop?
Sure. I guess SCM_RIGHTS are not too performance sensitive, but
that's a trivial cleanup...
> 3) unix_get_socket() probably ought to be static nowadays...
> 4) I wonder if in scan_inflight()/scan_children() we would be better
> off with explicit switch (by enum argument) instead of an indirect call.
Right.
> 5) do we really need UNIX_GC_MAYBE_CYCLE? Looks like the only
> real use is in inc_inflight_move_tail(), and AFAICS it could bloody well
> have been
> u->inflight++;
> if (u->inflight == 1) // just got from zero to non-zero
> list_move_tail(&u->link, &gc_candidates);
> The logics there is "we'd found a reference to something that still was
> a candidate for eviction in a reachable SCM_RIGHTS, so it's actually
> reachable and needs to be scanned (and removed from the set of candidates);
> move to the end of list, so that the main loop gets around to it".
> If it *was* past the cursor in the list, there's no need to move it; if
> we got past it, it must've had zero ->inflight (or we would've removed
> it from the set back when we got past it). Note that it's only called if
> UNIX_GC_CANDIDATE is set (i.e. if it's in the initial candidate set),
> so for this one ->inflight is guaranteed to mean the number of SCM_RIGHTS
> refs from outside of the current candidate set...
Right, makes sense.
> 6) unix_get_socket() looks like it might benefit from another
> FMODE bit; not sure where it's best set, though - the obvious way would
> be SOCK_GC_CARES in sock->flags, set by e.g. unix_create1(), with
> sock_alloc_file() propagating it into file->f_mode. Then unix_get_socket()
> would be able to bugger off with NULL for most of the references in
> SCM_RIGHTS, without looking into inode...
Yep, sounds good.
Thanks,
Miklos
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Al Viro @ 2019-02-07 13:31 UTC (permalink / raw)
To: Miklos Szeredi
Cc: Jens Axboe, Jann Horn, linux-aio, linux-block, Linux API, hch,
jmoyer, avi, linux-fsdevel
In-Reply-To: <20190207092253.GD19821@veci.piliscsaba.redhat.com>
On Thu, Feb 07, 2019 at 10:22:53AM +0100, Miklos Szeredi wrote:
> On Thu, Feb 07, 2019 at 04:00:59AM +0000, Al Viro wrote:
>
> > So in theory it would be possible to have
> > * thread A: sendmsg() has SCM_RIGHTS created and populated,
> > complete with file refcount and ->inflight increments implied,
> > at which point it gets preempted and loses the timeslice.
> > * thread B: gets to run and removes all references
> > from descriptor table it shares with thread A.
> > * on another CPU we have garbage collector triggered;
> > it determines the set of potentially unreachable unix_sock and
> > everything in our SCM_RIGHTS _is_ in that set, now that no
> > other references remain.
> > * on the first CPU, thread A regains the timeslice
> > and inserts its SCM_RIGHTS into queue. And it does contain
> > references to sockets from the candidate set of running
> > garbage collector, confusing the hell out of it.
>
> Reminds me: long time ago there was a bug report, and based on that I found a
> bug in MSG_PEEK handling (not confirmed to have fixed the reported bug). This
> fix, although pretty simple, got lost somehow. While unix gc code is in your
> head, can you please review and I'll resend through davem?
Umm... I think the bug is real (something that looks like an eviction
candidate, but actually is referenced from the reachable queue might
get peeked via that queue, then have _its_ queue modified via new
external reference, all between two passes over that queue, confusing the
fuck out of unix_gc()), but I think the fix is an overkill...
Am I right assuming that this queue-modifying operation is accept(), removing
an embryo unix_sock from the queue of listener and thus hiding SCM_RIGHTS in
_its_ queue from scan_children()?
Let me think of it a bit, OK? While we are at it, some questions from digging
through the current net/unix/garbage.c:
1) is there any need for ->inflight to be atomic? All accesses are under
unix_gc_lock, after all...
2) pumping unix_gc on each sodding reference in SCM_RIGHTS (within
unix_notinflight()/unix_inflight()) looks atrocious... wouldn't it be better to
hold it over that loop?
3) unix_get_socket() probably ought to be static nowadays...
4) I wonder if in scan_inflight()/scan_children() we would be better
off with explicit switch (by enum argument) instead of an indirect call.
5) do we really need UNIX_GC_MAYBE_CYCLE? Looks like the only
real use is in inc_inflight_move_tail(), and AFAICS it could bloody well
have been
u->inflight++;
if (u->inflight == 1) // just got from zero to non-zero
list_move_tail(&u->link, &gc_candidates);
The logics there is "we'd found a reference to something that still was
a candidate for eviction in a reachable SCM_RIGHTS, so it's actually
reachable and needs to be scanned (and removed from the set of candidates);
move to the end of list, so that the main loop gets around to it".
If it *was* past the cursor in the list, there's no need to move it; if
we got past it, it must've had zero ->inflight (or we would've removed
it from the set back when we got past it). Note that it's only called if
UNIX_GC_CANDIDATE is set (i.e. if it's in the initial candidate set),
so for this one ->inflight is guaranteed to mean the number of SCM_RIGHTS
refs from outside of the current candidate set...
6) unix_get_socket() looks like it might benefit from another
FMODE bit; not sure where it's best set, though - the obvious way would
be SOCK_GC_CARES in sock->flags, set by e.g. unix_create1(), with
sock_alloc_file() propagating it into file->f_mode. Then unix_get_socket()
would be able to bugger off with NULL for most of the references in
SCM_RIGHTS, without looking into inode...
Comments? IIRC, you'd done the last serious round of rewriting unix_gc()
and friends...
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Miklos Szeredi @ 2019-02-07 9:22 UTC (permalink / raw)
To: Al Viro
Cc: Jens Axboe, Jann Horn, linux-aio, linux-block, Linux API, hch,
jmoyer, avi, linux-fsdevel
In-Reply-To: <20190207040058.GW2217@ZenIV.linux.org.uk>
On Thu, Feb 07, 2019 at 04:00:59AM +0000, Al Viro wrote:
> So in theory it would be possible to have
> * thread A: sendmsg() has SCM_RIGHTS created and populated,
> complete with file refcount and ->inflight increments implied,
> at which point it gets preempted and loses the timeslice.
> * thread B: gets to run and removes all references
> from descriptor table it shares with thread A.
> * on another CPU we have garbage collector triggered;
> it determines the set of potentially unreachable unix_sock and
> everything in our SCM_RIGHTS _is_ in that set, now that no
> other references remain.
> * on the first CPU, thread A regains the timeslice
> and inserts its SCM_RIGHTS into queue. And it does contain
> references to sockets from the candidate set of running
> garbage collector, confusing the hell out of it.
Reminds me: long time ago there was a bug report, and based on that I found a
bug in MSG_PEEK handling (not confirmed to have fixed the reported bug). This
fix, although pretty simple, got lost somehow. While unix gc code is in your
head, can you please review and I'll resend through davem?
Thanks,
Miklos
---
From: Miklos Szeredi <mszeredi@redhat.com>
Subject: af_unix: fix garbage collect vs. MSG_PEEK
Gc assumes that in-flight sockets that don't have an external ref can't
gain one while unix_gc_lock is held. That is true because
unix_notinflight() will be called before detaching fds, which takes
unix_gc_lock.
Only MSG_PEEK was somehow overlooked. That one also clones the fds, also
keeping them in the skb. But through MSG_PEEK an external reference can
definitely be gained without ever touching unix_gc_lock.
This patch adds unix_gc_barrier() that waits for a garbage collect run to
finish (if there is one), before actually installing the peeked in-flight
files to file descriptors. This prevents problems from a pure in-flight
socket having its buffers modified while the garbage collect is taking
place.
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Cc: <stable@vger.kernel.org>
---
include/net/af_unix.h | 1 +
net/unix/af_unix.c | 15 +++++++++++++--
net/unix/garbage.c | 6 ++++++
3 files changed, 20 insertions(+), 2 deletions(-)
--- a/include/net/af_unix.h
+++ b/include/net/af_unix.h
@@ -12,6 +12,7 @@ void unix_inflight(struct user_struct *u
void unix_notinflight(struct user_struct *user, struct file *fp);
void unix_gc(void);
void wait_for_unix_gc(void);
+void unix_gc_barrier(void);
struct sock *unix_get_socket(struct file *filp);
struct sock *unix_peer_get(struct sock *sk);
--- a/net/unix/af_unix.c
+++ b/net/unix/af_unix.c
@@ -1547,6 +1547,17 @@ static int unix_attach_fds(struct scm_co
return 0;
}
+static void unix_peek_fds(struct scm_cookie *scm, struct sk_buff *skb)
+{
+ scm->fp = scm_fp_dup(UNIXCB(skb).fp);
+ /*
+ * During garbage collection it is assumed that in-flight sockets don't
+ * get a new external reference. So we need to wait until current run
+ * finishes.
+ */
+ unix_gc_barrier();
+}
+
static int unix_scm_to_skb(struct scm_cookie *scm, struct sk_buff *skb, bool send_fds)
{
int err = 0;
@@ -2171,7 +2182,7 @@ static int unix_dgram_recvmsg(struct soc
sk_peek_offset_fwd(sk, size);
if (UNIXCB(skb).fp)
- scm.fp = scm_fp_dup(UNIXCB(skb).fp);
+ unix_peek_fds(&scm, skb);
}
err = (flags & MSG_TRUNC) ? skb->len - skip : size;
@@ -2412,7 +2423,7 @@ static int unix_stream_read_generic(stru
/* It is questionable, see note in unix_dgram_recvmsg.
*/
if (UNIXCB(skb).fp)
- scm.fp = scm_fp_dup(UNIXCB(skb).fp);
+ unix_peek_fds(&scm, skb);
sk_peek_offset_fwd(sk, chunk);
--- a/net/unix/garbage.c
+++ b/net/unix/garbage.c
@@ -267,6 +267,12 @@ void wait_for_unix_gc(void)
wait_event(unix_gc_wait, gc_in_progress == false);
}
+void unix_gc_barrier(void)
+{
+ spin_lock(&unix_gc_lock);
+ spin_unlock(&unix_gc_lock);
+}
+
/* The external entry point: unix_gc() */
void unix_gc(void)
{
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
* Re: [PATCH 16/32] x86/vdso: Generate vdso{,32}-timens.lds
From: Rasmus Villemoes @ 2019-02-07 8:31 UTC (permalink / raw)
To: Dmitry Safonov, linux-kernel
Cc: Adrian Reber, Andrei Vagin, Andrei Vagin, Andy Lutomirski,
Andy Tucker, Arnd Bergmann, Christian Brauner, Cyrill Gorcunov,
Dmitry Safonov, Eric W. Biederman, H. Peter Anvin, Ingo Molnar,
Jeff Dike, Oleg Nesterov, Pavel Emelyanov, Shuah Khan,
Thomas Gleixner, containers, criu, linux-api, x86
In-Reply-To: <20190206001107.16488-17-dima@arista.com>
On 06/02/2019 01.10, Dmitry Safonov wrote:
> As it has been discussed on timens RFC, adding a new conditional branch
> `if (inside_time_ns)` on VDSO for all processes is undesirable.
> It will add a penalty for everybody as branch predictor may mispredict
> the jump. Also there are instruction cache lines wasted on cmp/jmp.
>
> Those effects of introducing time namespace are very much unwanted
> having in mind how much work have been spent on micro-optimisation
> vdso code.
>
> Addressing those problems, there are two versions of VDSO's .so:
> for host tasks (without any penalty) and for processes inside of time
> namespace with clk_to_ns() that subtracts offsets from host's time.
>
> Unfortunately, to allow changing VDSO VMA on a running process,
> the entry points to VDSO should have the same offsets (addresses).
> That's needed as i.e. application that calls setns() may have already
> resolved VDSO symbols in GOT/PLT.
These (14-19, if I'm reading them right) seems to add quite a lot of
complexity and fragility to the build, and other architectures would
probably have to add something similar to their vdso builds.
I'm wondering why not make the rule be that a timens takes effect on
next execve?
Rasmus
^ permalink raw reply
* Re: [PATCH 05/32] timerfd/timens: Take into account ns clock offsets
From: Andrei Vagin @ 2019-02-07 6:38 UTC (permalink / raw)
To: Cyrill Gorcunov
Cc: Dmitry Safonov, linux-kernel, Adrian Reber, Andrei Vagin,
Andy Lutomirski, Andy Tucker, Arnd Bergmann, Christian Brauner,
Dmitry Safonov, Eric W. Biederman, H. Peter Anvin, Ingo Molnar,
Jeff Dike, Oleg Nesterov, Pavel Emelyanov, Shuah Khan,
Thomas Gleixner, containers, criu, linux-api, x86
In-Reply-To: <20190206085203.GB25251@uranus>
On Wed, Feb 06, 2019 at 11:52:03AM +0300, Cyrill Gorcunov wrote:
> On Wed, Feb 06, 2019 at 12:10:39AM +0000, Dmitry Safonov wrote:
> > From: Andrei Vagin <avagin@gmail.com>
> >
> > Make timerfd respect timens offsets.
> > Provide two helpers timens_clock_to_host() timens_clock_from_host() that
> > are useful to wire up timens to different kernel subsystems.
> > Following patches will use timens_clock_from_host(), added here for
> > completeness.
> >
> > Signed-off-by: Andrei Vagin <avagin@openvz.org>
> > Co-developed-by: Dmitry Safonov <dima@arista.com>
> > Signed-off-by: Dmitry Safonov <dima@arista.com>
> > ---
> > fs/timerfd.c | 16 +++++++++++-----
> > 1 file changed, 11 insertions(+), 5 deletions(-)
> >
> > diff --git a/fs/timerfd.c b/fs/timerfd.c
> > index 803ca070d42e..c7ae1e371912 100644
> > --- a/fs/timerfd.c
> > +++ b/fs/timerfd.c
> > @@ -26,6 +26,7 @@
> > #include <linux/syscalls.h>
> > #include <linux/compat.h>
> > #include <linux/rcupdate.h>
> > +#include <linux/time_namespace.h>
> >
> > struct timerfd_ctx {
> > union {
> > @@ -433,22 +434,27 @@ SYSCALL_DEFINE2(timerfd_create, int, clockid, int, flags)
> > }
> >
> > static int do_timerfd_settime(int ufd, int flags,
> > - const struct itimerspec64 *new,
> > + struct itimerspec64 *new,
> > struct itimerspec64 *old)
> > {
> > struct fd f;
> > struct timerfd_ctx *ctx;
> > int ret;
> >
> > - if ((flags & ~TFD_SETTIME_FLAGS) ||
> > - !itimerspec64_valid(new))
> > - return -EINVAL;
>
> Please don't defer this early test of a @flags value. Otherwise
> if @flags is invalid you continue fget/put/clock-to-host even
> if result will be dropped out then.
Cyrill, you are right. I moved this check together with
itimerspec64_valid(). The idea was to call itimerspec64_valid() after
applying clock offsets but for that, we need to know clockid.
Let's wait a bit for other comments to this patch set and then we will
fix all things what will be found.
Thanks,
Andrei
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Al Viro @ 2019-02-07 4:05 UTC (permalink / raw)
To: Jens Axboe
Cc: Jann Horn, linux-aio, linux-block, Linux API, hch, jmoyer, avi,
linux-fsdevel
In-Reply-To: <608c1102-5818-e38c-cfc7-b1cec5a1ecb4@kernel.dk>
On Wed, Feb 06, 2019 at 10:56:41AM -0700, Jens Axboe wrote:
> On 2/5/19 6:01 PM, Al Viro wrote:
> > On Tue, Feb 05, 2019 at 05:27:29PM -0700, Jens Axboe wrote:
> >
> >> This should be better, passes some basic testing, too:
> >>
> >> http://git.kernel.dk/cgit/linux-block/commit/?h=io_uring&id=01a93aa784319a02ccfa6523371b93401c9e0073
> >>
> >> Verified that we're grabbing the right refs, and don't hold any
> >> ourselves. For the file registration, forbid registration of the
> >> io_uring fd, as that is pointless and will introduce a loop regardless
> >> of fd passing.
> >
> > *shrug*
> >
> > So pass it to AF_UNIX socket and register _that_ - does't change the
> > underlying problem.
>
> Maybe I'm being dense here, but it's an f_op match. Should catch a
> passed fd as well, correct?
f_op match on _what_?
> With that, how can there be a loop?
io_uring_fd = ....
socketpair(PF_UNIX, SOCK_STREAM, 0, sock_fds);
register sock_fds[0] and sock_fds[1] to io_uring_fd
send SCM_RIGHTS datagram with io_uring_fd to sock_fds[0]
close sock_fds[0], sock_fds[1] and io_uring_fd
And there's your unreachable loop.
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
* Re: [PATCH 13/18] io_uring: add file set registration
From: Al Viro @ 2019-02-07 4:00 UTC (permalink / raw)
To: Jens Axboe
Cc: Jann Horn, linux-aio, linux-block, Linux API, hch, jmoyer, avi,
linux-fsdevel
In-Reply-To: <8f124de2-d6da-d656-25e4-b4d9e58f880e@kernel.dk>
On Wed, Feb 06, 2019 at 06:41:00AM -0700, Jens Axboe wrote:
> On 2/5/19 5:56 PM, Al Viro wrote:
> > On Tue, Feb 05, 2019 at 12:08:25PM -0700, Jens Axboe wrote:
> >> Proof is in the pudding, here's the main commit introducing io_uring
> >> and now wiring it up to the AF_UNIX garbage collection:
> >>
> >> http://git.kernel.dk/cgit/linux-block/commit/?h=io_uring&id=158e6f42b67d0abe9ee84886b96ca8c4b3d3dfd5
> >>
> >> How does that look?
> >
> > In a word - wrong. Some theory: garbage collector assumes that there is
> > a subset of file references such that
> > * for all files with such references there's an associated unix_sock.
> > * all such references are stored in SCM_RIGHTS datagrams that can be
> > found by the garbage collector (currently: for data-bearing AF_UNIX sockets -
> > queued SCM_RIGHTS datagrams, for listeners - SCM_RIGHTS datagrams sent via
> > yet-to-be-accepted connections).
> > * there is an efficient way to count those references for given file
> > (->inflight of the corresponding unix_sock).
> > * removal of those references would render the graph acyclic.
> > * file can _NOT_ be subject to syscalls unless there are references
> > to it outside of that subset.
>
> IOW, we cannot use fget() for registering files, and we still need fget/fput
> in the fast path to retain safe use of the file. If I'm understanding you
> correctly?
No. *ALL* references (inflight and not) are the same for file->f_count.
unix_inflight() does not grab a new reference to file; it only says that
reference passed to it by the caller is now an in-flight one.
OK, braindump time:
Lifetime for struct file is controlled by a simple refcount. Destructor
(__fput() and ->release() called from it) is called once the counter hits
zero. Which is fine, except for the situations when some struct file
references are pinned down in structures reachable only via our struct file
instance.
Each file descriptor counts as a reference. IOW, dup() will increment
the refcount by 1, close() will decrement it, fork() will increment it
by the number of descriptors in your descriptor table refering to this
struct file, desctruction of descriptor table on exit() will decrement
by the same amount, etc.
Syscalls like read() and friends turn descriptor(s) into struct file
references. If descriptor table is shared, that counts as a new reference
that must be dropped in the end of syscall. If it's not shared, we are
guaranteed that the reference in descriptor table will stay around until
the end of syscall, so we may use it without bumping the file refcount.
That's the difference between fget() and fdget() - the former will
bump the refcount, the latter will try to avoid that. Of course, if
we do not intend to drop the reference we'd acquired by the end of
syscall, we want fget() - fdget() is for transient references only.
Descriptor tables (struct files) *can* be shared; several processes
(usually - threads that share VM as well, but that's not necessary)
may be working with the same instance of struct files, so e.g. open()
in one of them is seen by the others. The same goes for close(),
dup(), dup2(), etc.
That makes for an interesting corner case - what if two threads happen
to share a descriptor table and we have close(fd) in one of them
in the middle of read(fd, ..., ...) in another? That's one aread where
Unices differ - one variant is to abort read(), another - to have close()
wait for read() to finish, etc. What we do is
* close() succeeds immediately; the reference is removed from
descriptor table and dropped.
* if close(fd) has happened before read(fd, ...) has converted
fd to struce file reference, read() will get -EBADF.
* otherwise, read() proceeds unmolested; the reference it has
acquired is dropped in the end of syscall. If that's the last reference
to struct file, struct file will get shut down at that point.
clone(2) will have the child sharing descriptor table of parent if
CLONE_FILES is in the flags. Note that in this case struct file
refcounts are not modified at all - no new references to files are
created. Without CLONE_FILES it's the same as fork() - an independent
copy of descriptor table is created and populated by copies of references
to files, each bumping file's refcount.
unshare(2) with CLONE_FILES in flags will get a copy of descriptor table
(same as done on fork(), etc.) and switch to using it; the old reference
is dropped (note: it'll only bother with that if descriptor table used
to be shared in the first place - if we hold the only reference to
descriptor table, we'll just keep using it).
execve(2) does almost the same - if descriptor table used to be shared,
it will switch to a new copy first; in case of success the reference
to original is dropped, in case of failure we revert to original and
drop the copy. Note that handling of close-on-exec is done in the _copy_
- the original is unaffected, so failing execve() does not disrupt the
descriptor table.
exit(2) will drop the reference to descriptor table. When the last
reference is dropped, all file references are removed from it (and dropped).
The thread's pointer to descriptor table (current->files) is never
modified by other thread; something like ls /proc/<pid>/fd will fetch
it, so stores need to be protected (by task_lock(current)), but
the only the thread itself can do them.
Note that while extra references to descriptor table can appear at any
time (/proc/<pid>/fd accesses, for example), such references may not
be used for modifications. In particular, you can't switch to another
thread's descriptor table, unless it had been yours at some earlier
point _and_ you've kept a reference to it.
That's about it for descriptor tables; that, by far, is the main case
of persistently held struct file references. Transient references
are grabbed by syscalls when they resolve descriptor to struct file *,
which ought to be done once per syscall _and_ reasonably early in
it. Unfortunately, that's not all - there are other persistent struct
file references.
The things like "LOOP_SET_FD grabs a reference to struct file and
stashes it in ->lo_backing_file" are reasonably simple - the reference
will be dropped later, either directly by LOOP_CLR_FD (if nothing else
held the damn thing open at the time) or later in lo_release().
Note that in the latter case it's possible to get "close() of
/dev/loop descriptor drops the last reference to it, triggering
bdput(), which happens to by the last thing that held block device
opened, which triggers lo_release(), which drops the reference to
underlying struct file (almost certainly the last one by that point)".
It's still not a problem - while we have the underlying struct file
pinned by something held by another struct file, the dependencies'
graph is acyclic, so plain refcounts we are using work fine.
The same goes for the things like e.g. ecryptfs opening an underlying
(encrypted) file on open() and dropping it when the last reference
to ecryptfs file is dropped - the only difference here is that the
underlying struct file is never appearing in _anyone's_ descriptor
tables.
However, in a couple of cases we do have something trickier.
Case 1: SCM_RIGHTS datagram can be sent to an AF_UNIX socket. That
converts the caller-supplied array of descriptors into an array of
struct file references, which gets attached to the packet we queue.
When the datagram is received, the struct file references are
moved into the descriptor table of recepient or, in case of error,
dropped. Note that sending some descriptors in an SCM_RIGHTS datagram
and closing them is perfectly legitimate - as soon as sendmsg(2)
returns you can go ahead and close the descriptors you've sent;
the references are already acquired and you don't need to wait for
the packet to be received.
That would still be simple, if not for the fact that there's nothing
to stop you from passing AF_UNIX sockets around the same way. In fact,
that has legitimate uses and, most of the time, doesn't cause any
complications at all. However, it is possible to get the situation
when
* struct file instances A and B are both AF_UNIX sockets.
* the only reference to A is in the SCM_RIGHTS packet that
sits in the receiving queue of B.
* the only reference to B is in the SCM_RIGHTS packet that
sits in the receiving queue of A.
That, of course, is where the pure refcounting of any kind will break.
SCM_RIGHTS datagram that contains the sole reference to A can't be
received without the recepient getting hold of a reference to B.
Which cannot happen until somebody manages to receive the SCM_RIGHTS
datagram containing the sole reference to B. Which cannot happen
until that somebody manages to get hold of a reference to A,
which cannot happen until the first SCM_RIGHTS datagram is
received.
Dropping the last reference to A would've discarded everything in
its receiving queue, including the SCM_RIGHTS that contains the
reference to B; however, that can't happen either - the other
SCM_RIGHTS datagram would have to be either received or discarded
first, etc.
Case 2: similar, with a bit of a twist. AF_UNIX socket used for
descriptor passing is normally set up by socket(), followed by
connect(). As soon as connect() returns, one can start sending.
Note that connect() does *NOT* wait for the recepient to call
accept() - it creates the object that will serve as low-level
part of the other end of connection (complete with received
packet queue) and stashes that object into the queue of *listener*
socket. Subsequent accept() fetches it from there and attaches
it to a new socket, completing the setup; in the meanwhile,
sending packets works fine. Once accept() is done, it'll see
the stuff you'd sent already in the queue of the new socket and
everything works fine.
If the listening socket gets closed without accept() having
been called, its queue is flushed, discarding all pending
connection attempts, complete with _their_ queues. Which is
the same effect as accept() + close(), so again, normally
everything just works.
However, consider the case when we have
* struct file instances A and B being AF_UNIX sockets.
* A is a listener
* B is an established connection, with the other end
yet to be accepted on A
* the only references to A and B are in an SCM_RIGHTS
datagram sent over by A.
That SCM_RIGHTS datagram could've been received, if somebody
had managed to call accept(2) on A and recvmsg(2) on the
socket created by that accept(2). But that can't happen
without that somebody getting hold of a reference to A in
the first place, which can't happen without having received
that SCM_RIGHTS datagram. It can't be discarded either,
since that can't happen without dropping the last reference
to A, which sits right in it.
The difference from the previous case is that there we had
A holds unix_sock of A
unix_sock of A holds SCM_RIGHTS with reference to B
B holds unix_sock of B
unix_sock of B holds SCM_RIGHTS with reference to A
and here we have
A holds unix_sock of A
unix_sock of A holds the packet with reference to embryonic
unix_sock created by connect()
that embryionic unix_sock holds SCM_RIGHTS with references
to A and B.
Dependency graph is different, but the problem is the same -
unreachable loops in it. Note that neither class of situations
would occur normally - in the best case it's "somebody had been
doing rather convoluted descriptor passing, but everyone involved
got hit with kill -9 at the wrong time; please, make sure nothing
leaks". That can happen, but a userland race (e.g. botched protocol
handling of some sort) or a deliberate abuse are much more likely.
Catching the loop creation is hard and paying for that every time
we do descriptor-passing would be a bad idea. Besides, the loop
per se is not fatal - e.g if in the second case the descriptor for
A had been kept around, close(accept()) would've cleaned everything
up. Which means that we need a garbage collector to deal with
the (rare) leaks.
Note that in both cases the leaks are caused by loops passing through
some SCM_RIGHTS datagrams that could never be received. So locating
those, removing them from the queues they sit in and then discarding
the suckers is enough to resolve the situation.
Furthermore, in both cases the loop passes through unix_sock of
something that got sent over in an SCM_RIGHTS datagram. So we
can do the following:
1) keep the count of references to struct file of AF_UNIX
socket held by SCM_RIGHTS (kept in unix_sock->inflight). Any
struct unix_sock instance without such references is not a part of
unreachable loop. Maintain the set of unix_sock that are not excluded
by that (i.e. the ones that have some of references from SCM_RIGHTS
instances). Note that we don't need to maintain those counts in
struct file - we care only about unix_sock here.
2) struct file of AF_UNIX socket with some references
*NOT* from SCM_RIGHTS is also not a part of unreachable loop.
3) for each unix_sock consider the following set of
SCM_RIGHTS: everything in queue of that unix_sock if it's
a non-listener and everything in queues of all embryonic unix_sock
in queue of a listener. Let's call those SCM_RIGHTS associated
with our unix_sock.
4) all SCM_RIGHTS associated with a reachable unix_sock
are reachable.
5) if some references to struct file of a unix_sock
are in reachable SCM_RIGHTS, it is reachable.
Garbage collector starts with calculating the set of potentially
unreachable unix_sock - the ones not excluded by (1,2).
No unix_sock instances outside of that set need to be considered.
If some unix_sock in that set has counter _not_ entirely covered
by SCM_RIGHTS associated with the elements of the set, we can
conclude that there are references to it in SCM_RIGHTS associated
with something outside of our set and therefore it is reachable
and can be removed from the set.
If that process converges to a non-empty set, we know that
everything left in that set is unreachable - all references
to their struct file come from _some_ SCM_RIGHTS datagrams
and all those SCM_RIGHTS datagrams are among those that can't
be received or discarded without getting hold of a reference
to struct file of something in our set.
Everything outside of that set is reachable, so taking the
SCM_RIGHTS with references to stuff in our set (all of
them to be found among those associated with elements of
our set) out of the queues they are in will break all
unreachable loops. Discarding the collected datagrams
will do the rest - file references in those will be
dropped, etc.
One thing to keep in mind here is the locking. What the garbage
collector relies upon is
* changes of ->inflight are serialized with respect to
it (on unix_gc_lock; increment done by unix_inflight(),
decrement - by unix_notinflight()).
* references cannot be extracted from SCM_RIGHTS datagrams
while the garbage collector is running (achieved by having
unix_notinflight() done before references out of SCM_RIGHTS)
* removal of SCM_RIGHTS associated with a socket can't
be done without a reference to that socket _outside_ of any
SCM_RIGHTS (automatically true).
* adding SCM_RIGHTS in the middle of garbage collection
is possible, but in that case it will contain no references to
anything in the initial candidate set.
The last one is delicate. SCM_RIGHTS creation has unix_inflight()
called for each reference we put there, so it's serialized wrt
unix_gc(); however, insertion into queue is *NOT* covered by that -
queue rescans are, but each queue has a lock of its own and they
are definitely not going to be held throughout the whole thing.
So in theory it would be possible to have
* thread A: sendmsg() has SCM_RIGHTS created and populated,
complete with file refcount and ->inflight increments implied,
at which point it gets preempted and loses the timeslice.
* thread B: gets to run and removes all references
from descriptor table it shares with thread A.
* on another CPU we have garbage collector triggered;
it determines the set of potentially unreachable unix_sock and
everything in our SCM_RIGHTS _is_ in that set, now that no
other references remain.
* on the first CPU, thread A regains the timeslice
and inserts its SCM_RIGHTS into queue. And it does contain
references to sockets from the candidate set of running
garbage collector, confusing the hell out of it.
That is avoided by a convoluted dance around the SCM_RIGHTS creation
and insertion - we use fget() to obtain struct file references,
then _duplicate_ them in SCM_RIGHTS (bumping a refcount for each, so
we are holding *two* references), do unix_inflight() on them, then
queue the damn thing, then drop each reference we got from fget().
That way everything refered to in that SCM_RIGHTS is going to have
extra struct file references (and thus be excluded from the initial
candidate set) until after it gets inserted into queue. In other
words, if it does appear in a queue between two passes, it's
guaranteed to contain no references to anything in the initial
canidate set.
End of braindump.
===================================================================
What you are about to add is *ANOTHER* kind of loops - references
to files in the "registered" set are pinned down by owning io_uring.
That would invalidate just about every assumption made the garbage
collector - even if you forbid to register io_uring itself, you
still can register both ends of AF_UNIX socket pair, then pass
io_uring in SCM_RIGHTS over that, then close all descriptors involved.
>From the garbage collector point of view all sockets have external
references, so there's nothing to collect. In fact those external
references are only reachable if you have a reachable reference
to io_uring, so we get a leak.
To make it work:
* have unix_sock created for each io_uring (as your code does)
* do *NOT* have unix_inflight() done at that point - it's
completely wrong there.
* file set registration becomes
* create and populate SCM_RIGHTS, with the same
fget()+grab an extra reference + unix_inflight() sequence.
Don't forget to have skb->destructor set to unix_destruct_scm
or equivalent thereof.
* remember UNIXCB(skb).fp - that'll give you your
array of struct file *, to use in lookups.
* queue it into your unix_sock
* do _one_ fput() for everything you've grabbed,
dropping one of two references you've taken.
* unregistering is simply skb_dequeue() + kfree_skb().
* in ->release() you do sock_release(); it'll do
everything you need (including unregistering the set, etc.)
The hairiest part is the file set registration, of course -
there's almost certainly a helper or two buried in that thing;
simply exposing all the required net/unix/af_unix.c bits is
ucking fugly.
I'm not sure what you propose for non-registered descriptors -
_any_ struct file reference that outlives the return from syscall
stored in some io_uring-attached data structure is has exact same
loop (and leak) problem. And if you mean to have it dropped before
return from syscall, I'm afraid I don't follow you. How would
that be done?
Again, "io_uring descriptor can't be used in those requests" does
not help at all - use a socket instead, pass the io_uring fd over
it in SCM_RIGHTS and you are back to square 1.
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox