From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F3AFC388F7 for ; Sat, 31 Oct 2020 11:44:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0A28620791 for ; Sat, 31 Oct 2020 11:44:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1604144652; bh=1e27utaQ9tJp9kp1bR9gN77QNOeihN6Zfi5+HFYj4K8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=qZ40wGd8WSPuvknAy05mbHwu76iYCDNe0HFBEuj2VsNMKa52d7LWsfS2EQxAik4NY ZupF5LXdlths9Fha6rq673R6s8Ts0wf/JZErPTnNrk7hZlnKqm/Ek7D2IX6iFhC1VB 4UDCpLNk5nELombR+6rq9DMhXkP3RsZM7DSw6Gu4= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728126AbgJaLoL (ORCPT ); Sat, 31 Oct 2020 07:44:11 -0400 Received: from mail.kernel.org ([198.145.29.99]:44246 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727065AbgJaLoD (ORCPT ); Sat, 31 Oct 2020 07:44:03 -0400 Received: from localhost (83-86-74-64.cable.dynamic.v4.ziggo.nl [83.86.74.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0BFC6205F4; Sat, 31 Oct 2020 11:44:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1604144642; bh=1e27utaQ9tJp9kp1bR9gN77QNOeihN6Zfi5+HFYj4K8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ybizDPp3zQwmQgN97BK/JjjRMYR2ALQAoqhvTMUN5ahN/8QMPkOQCj40U+G8tElNW d857d/0Uv9BEZi0r7k6GjCpxu67haL1/jOPwWLQcvMjdCKqJA7FpyflSI4g2TC3lzE hT2AyE3wCw/PWTFBIB/DhRIu10VAm4stTu/uX5Is= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Pavel Begunkov , Jens Axboe Subject: [PATCH 5.9 08/74] io_uring: enable task/files specific overflow flushing Date: Sat, 31 Oct 2020 12:35:50 +0100 Message-Id: <20201031113500.441612345@linuxfoundation.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201031113500.031279088@linuxfoundation.org> References: <20201031113500.031279088@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Jens Axboe commit e6c8aa9ac33bd7c968af7816240fc081401fddcd upstream. This allows us to selectively flush out pending overflows, depending on the task and/or files_struct being passed in. No intended functional changes in this patch. Reviewed-by: Pavel Begunkov Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman --- fs/io_uring.c | 41 +++++++++++++++++++++++++---------------- 1 file changed, 25 insertions(+), 16 deletions(-) --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -1344,12 +1344,24 @@ static void io_cqring_mark_overflow(stru } } +static inline bool io_match_files(struct io_kiocb *req, + struct files_struct *files) +{ + if (!files) + return true; + if (req->flags & REQ_F_WORK_INITIALIZED) + return req->work.files == files; + return false; +} + /* Returns true if there are no backlogged entries after the flush */ -static bool io_cqring_overflow_flush(struct io_ring_ctx *ctx, bool force) +static bool io_cqring_overflow_flush(struct io_ring_ctx *ctx, bool force, + struct task_struct *tsk, + struct files_struct *files) { struct io_rings *rings = ctx->rings; + struct io_kiocb *req, *tmp; struct io_uring_cqe *cqe; - struct io_kiocb *req; unsigned long flags; LIST_HEAD(list); @@ -1368,13 +1380,16 @@ static bool io_cqring_overflow_flush(str ctx->cq_overflow_flushed = 1; cqe = NULL; - while (!list_empty(&ctx->cq_overflow_list)) { + list_for_each_entry_safe(req, tmp, &ctx->cq_overflow_list, compl.list) { + if (tsk && req->task != tsk) + continue; + if (!io_match_files(req, files)) + continue; + cqe = io_get_cqring(ctx); if (!cqe && !force) break; - req = list_first_entry(&ctx->cq_overflow_list, struct io_kiocb, - compl.list); list_move(&req->compl.list, &list); if (cqe) { WRITE_ONCE(cqe->user_data, req->user_data); @@ -1988,7 +2003,7 @@ static unsigned io_cqring_events(struct if (noflush && !list_empty(&ctx->cq_overflow_list)) return -1U; - io_cqring_overflow_flush(ctx, false); + io_cqring_overflow_flush(ctx, false, NULL, NULL); } /* See comment at the top of this file */ @@ -6489,7 +6504,7 @@ static int io_submit_sqes(struct io_ring /* if we have a backlog and couldn't flush it all, return BUSY */ if (test_bit(0, &ctx->sq_check_overflow)) { if (!list_empty(&ctx->cq_overflow_list) && - !io_cqring_overflow_flush(ctx, false)) + !io_cqring_overflow_flush(ctx, false, NULL, NULL)) return -EBUSY; } @@ -7993,7 +8008,7 @@ static void io_ring_exit_work(struct wor */ do { if (ctx->rings) - io_cqring_overflow_flush(ctx, true); + io_cqring_overflow_flush(ctx, true, NULL, NULL); io_iopoll_try_reap_events(ctx); } while (!wait_for_completion_timeout(&ctx->ref_comp, HZ/20)); io_ring_ctx_free(ctx); @@ -8013,7 +8028,7 @@ static void io_ring_ctx_wait_and_kill(st /* if we failed setting up the ctx, we might not have any rings */ if (ctx->rings) - io_cqring_overflow_flush(ctx, true); + io_cqring_overflow_flush(ctx, true, NULL, NULL); io_iopoll_try_reap_events(ctx); idr_for_each(&ctx->personality_idr, io_remove_personalities, ctx); @@ -8069,12 +8084,6 @@ static bool io_match_link(struct io_kioc return false; } -static inline bool io_match_files(struct io_kiocb *req, - struct files_struct *files) -{ - return (req->flags & REQ_F_WORK_INITIALIZED) && req->work.files == files; -} - static bool io_match_link_files(struct io_kiocb *req, struct files_struct *files) { @@ -8365,7 +8374,7 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned ret = 0; if (ctx->flags & IORING_SETUP_SQPOLL) { if (!list_empty_careful(&ctx->cq_overflow_list)) - io_cqring_overflow_flush(ctx, false); + io_cqring_overflow_flush(ctx, false, NULL, NULL); if (flags & IORING_ENTER_SQ_WAKEUP) wake_up(&ctx->sqo_wait); submitted = to_submit;