From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id BDFE8381B1F for ; Fri, 10 Apr 2026 11:14:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775819648; cv=none; b=mJlsmGducurIaot6lSHzAFRE7tt7qcMKm8PQaui1auao3V+BrkPESp7+6al1IE0RD5zd4tMOw1Yb0Pc4v044GmXrvALtOPtBVQVQ5+uqz5ztXVTu71cqPeiyT6BreAqPZ+DDkgNaDyuKVLs8MbLD1lVmqwjhJgWynb+2Zo2j2tc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775819648; c=relaxed/simple; bh=VseHsv7XV3fv+gy3fkPYuQnfm8odatXf0Pg9LU7i2ms=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=npvu+f0aakwDbhVEthF63SsQ9THr4i9ZBc2B+IAT4n3GWO/Ic7dW1qGEh37/F0vzKkX4G/a4aDx+2g3tnQzpkcX52lVk/EeZ78d4eW9noPhSgNJw6d4BIibSssxbKsILFfpRzx+Y4B6XzHhkpOiORBkB8uGjsNI4BQzX6gaMTfE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=dwP5i798; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="dwP5i798" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 772872682; Fri, 10 Apr 2026 04:14:00 -0700 (PDT) Received: from [192.168.77.1] (usa-sjc-mx-foss1.foss.arm.com [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E42143FAF5; Fri, 10 Apr 2026 04:14:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1775819646; bh=VseHsv7XV3fv+gy3fkPYuQnfm8odatXf0Pg9LU7i2ms=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=dwP5i798bAmwXQIi/frf1Ssc+If+uvOApxz+rz55ygYCZCJzLTbPJMCrRmJ9hoBWq I5cM/iN9nnEvEHwtVMdhslPnlzeO9nUMFxEsrUiE9OUxeeOsNPlcQhu15JHnhIMY1N Qn0w+AVdBliE6QoXUAhvFKY3GAcSCBEbwFJff94E= Message-ID: <3ed0139b-9e34-491d-80cf-e818fadadd06@arm.com> Date: Fri, 10 Apr 2026 12:13:55 +0100 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [Question] Sched: Severe scheduling latency (>10s) observed on kernel 6.12 with specific workload To: Xuewen Yan Cc: Vincent Guittot , John Stultz , Peter Zijlstra , Ingo Molnar , Juri Lelli , Steven Rostedt , Benjamin Segall , Mel Gorman , Valentin Schneider , linux-kernel , =?UTF-8?B?546L56eRIChLZSBXYW5nKQ==?= , Xuewen Yan , "hongyu.jin@unisoc.com" , guohua.yan@unisoc.com References: <1c0e5a70-22b7-4cbd-8ca9-0a7da27ed7a9@arm.com> <89c464b5-908d-4817-87f4-d8cb905495a4@arm.com> Content-Language: en-GB From: Dietmar Eggemann In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 08.04.26 03:50, Xuewen Yan wrote: > Hi Dietmar and Vincent, > >> I tried to recreate your env as much as possible on qemu and ran your >> rt-app file but I can't spot anything suspicious either. This is with >> defconfig and cgroupv2. > > Could you please try the following configuration? > To rule out Android's influence, I created two new cgroups: > foreground_test and background_test. > I then placed only rt-app threads into these groups. Even with this > setup, we can still observe high scheduling latency for tasks in > foreground_test. Is this still on an Android (vendor hooks, etc.) or mainline 6.12.58 kernel/device? > { > "tasks" : { > "t0" : { > "instance" : 40, > "priority" : 0, > "cpus" : [ 0, 1, 2, 3 ], > "taskgroup" : "/background_test", > "loop" : -1, > "run" : 200, > "sleep" : 50 > }, > "t1" : { > "instance" : 2, > "priority" : 19, > "cpus" : [ 0, 1, 2, 3 ], > "taskgroup" : "/foreground_test", > "loop" : -1, > "run" : 60000, > "sleep" : 100000 > }, > "t2" : { > "instance" : 2, > "priority" : 10, > "cpus" : [ 0, 1, 2, 3 ], > "taskgroup" : "/foreground_test", > "loop" : -1, > "run" : 5000, > "sleep" : 100000 > } > } > } With your rt-app file and moving the tasks into cgroupv2 taskgroups manually: t0-0 679 0 > /sys/fs/cgroup/A t0-1 680 0 > /sys/fs/cgroup/A t0-2 681 0 > /sys/fs/cgroup/A ... t0-37 716 0 > /sys/fs/cgroup/A t0-38 717 0 > /sys/fs/cgroup/A t0-39 718 0 > /sys/fs/cgroup/A t1-40 719 19 > /sys/fs/cgroup/B t1-41 720 19 > /sys/fs/cgroup/B t2-42 721 10 > /sys/fs/cgroup/B t2-43 722 10 > /sys/fs/cgroup/B 10 highest wu_lat values on Arm64 qemu (-accel hvf): v6.6 0.564159000 t2-43:684 0.306418000 t2-42:683 0.237134000 t2-43:684 0.166982000 t2-42:683 0.166674000 t2-42:683 0.161856000 t2-42:683 0.098879000 t2-43:684 0.097746000 t2-43:684 0.083329000 t2-42:683 0.082943000 t2-42:683 6.12.58 0.368566000 t2-43:939 0.228139000 t2-42:938 0.212454000 t2-43:939 0.207144000 t2-42:938 0.177373000 t2-43:939 0.148268000 t2-42:938 0.147619000 t2-42:938 0.125988000 t2-43:939 0.091564000 t2-42:938 0.088160000 t2-42:938 tip sched/core (7.0.0-rc6-00050-g985215804dcb) 0.395585000 t2-43:697 0.203889000 t2-43:697 0.101130000 t2-42:696 0.098782000 t2-42:696 0.084523000 t2-43:697 0.033895000 t2-42:696 0.031881000 t2-43:697 0.021958000 t2-42:696 0.018123000 t2-42:696 0.013132000 t0-7:661 Could you specify which tasks had those > 1s wu_lat values?