Re: single aio thread is migrated crazily by scheduler

[Srikar Dronamraju <srikar@linux.vnet.ibm.com>]

* Dave Chinner <david@fromorbit.com> [2019-11-16 10:40:05]:

> On Fri, Nov 15, 2019 at 03:08:43PM +0800, Ming Lei wrote:
> > On Fri, Nov 15, 2019 at 03:56:34PM +1100, Dave Chinner wrote:
> > > On Fri, Nov 15, 2019 at 09:08:24AM +0800, Ming Lei wrote:
> > I can reproduce the issue with 4k block size on another RH system, and
> > the login info of that system has been shared to you in RH BZ.
> > 
> > 1)
> 
> Almost all the fio task migrations are coming from migration/X
> kernel threads. i.e it's the scheduler active balancing that is
> causing the fio thread to bounce around.
> 

Can we try with the below patch.

-- 
Thanks and Regards
Srikar Dronamraju

--->8-----------------------------8<----------------------------------
From 9687c1447532558aa564bd2e471b7987d6bda70f Mon Sep 17 00:00:00 2001
From: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Date: Tue, 2 Jul 2019 16:38:29 -0500
Subject: [PATCH] sched/fair: Avoid active balance on small load imbalance

Skip active load balance when destination CPU is busy and the imbalance
is small and fix_small_imabalance is unable to calculate minor
imbalance. Its observed that active load balances can lead to ping-pong
of tasks between two CPUs.

Signed-off-by: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
---
 kernel/sched/fair.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 3599bdcab395..0db380c8eb6c 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7094,6 +7094,7 @@ enum group_type {
 #define LBF_SOME_PINNED	0x08
 #define LBF_NOHZ_STATS	0x10
 #define LBF_NOHZ_AGAIN	0x20
+#define LBF_SMALL_IMBL	0x40
 
 struct lb_env {
 	struct sched_domain	*sd;
@@ -8386,6 +8387,8 @@ void fix_small_imbalance(struct lb_env *env, struct sd_lb_stats *sds)
 	/* Move if we gain throughput */
 	if (capa_move > capa_now)
 		env->imbalance = busiest->load_per_task;
+	else if (env->idle == CPU_NOT_IDLE)
+		env->flags |= LBF_SMALL_IMBL;
 }
 
 /**
@@ -8466,7 +8469,7 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *s
 	 * moved
 	 */
 	if (env->imbalance < busiest->load_per_task)
-		return fix_small_imbalance(env, sds);
+		fix_small_imbalance(env, sds);
 }
 
 /******* find_busiest_group() helpers end here *********************/
@@ -8732,6 +8735,13 @@ static int need_active_balance(struct lb_env *env)
 	if (voluntary_active_balance(env))
 		return 1;
 
+	/*
+	 * Destination CPU is not idle and fix_small_imbalance is unable
+	 * to calculate even minor imbalances, skip active balance.
+	 */
+	if (env->flags & LBF_SMALL_IMBL)
+		return 0;
+
 	return unlikely(sd->nr_balance_failed > sd->cache_nice_tries+2);
 }
 
-- 
2.18.1