From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 929291F4E51 for ; Tue, 7 Jan 2025 16:25:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736267138; cv=none; b=dgCB+Ovb8YKQK6azAsrlA8fPKInVkRRTMPoirO0LY9aqkE3NKJ3KdZzLaYvm5tjGlF3Kowy3pNgjTsWKOh0Fu54RqqiXMqt7VWuVNhwHmzQ0VhRUwyRJwMhUHAPsQqXTLvL6CrJGIXvO3KCQD5bAzeRslWZDXq/ojEw8Zex6NlU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736267138; c=relaxed/simple; bh=OjhrNpU6cMjA4qdo/crJY/Hez0GMp9LS3pZfm52qM48=; h=From:To:Cc:References:In-Reply-To:Subject:Date:Message-ID: MIME-Version:Content-Type; b=t+KmevVKoPOWvoSItitctJvOJhuA4FbzX06Dudggm8rCW3ROnHcZKir85q8fdtSkkZqpbb03G06MGAynCvvLH5gbTcfd2/soiE3A7l5fw65jWhMQK8mP0sg1MxHQrFL27MFv8qneGGko8+Ouork8FUxOKIZ6mVoDkSSm+YHtR6g= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=telus.net; spf=pass smtp.mailfrom=telus.net; dkim=pass (2048-bit key) header.d=telus.net header.i=@telus.net header.b=h8THn94i; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=telus.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=telus.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=telus.net header.i=@telus.net header.b="h8THn94i" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-218c8aca5f1so39087785ad.0 for ; Tue, 07 Jan 2025 08:25:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telus.net; s=google; t=1736267136; x=1736871936; darn=vger.kernel.org; h=thread-index:content-language:content-transfer-encoding :mime-version:message-id:date:subject:in-reply-to:references:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=peAHr/1/p3X4gbVHd+S39qeQdzNdcR8zF0XPSGcv9KA=; b=h8THn94iPG4tsEjoGGprPoZgO401ZxtR3OVQUAF0tRkCWj+mMQBsmvT7oUYau9Tdar 29IP3DZyNkmij9h7Crle2qEC4lb+g4EtBc7B4sHgHwO7cfbYaGmlDgUAPSpiZouRjCa8 DIE7AUa7EmCzr4xuYrUGihxeO3dRdu7iqPuTynHhIg8nWyJpts1l2IdFhybqscNN4Nx9 KDsG9VesudPppjhvdYz1HUis8vagbPGmQtQJL41PX2tM8x/CmbbNJ3fhngbLYkTbsotM jdQhDkL+t/T1bn7+aWedZ2+krrhszi6kd2oJ/SGfGyU+l95KIkveG5QbZpGe5sEFqut/ HPjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736267136; x=1736871936; h=thread-index:content-language:content-transfer-encoding :mime-version:message-id:date:subject:in-reply-to:references:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=peAHr/1/p3X4gbVHd+S39qeQdzNdcR8zF0XPSGcv9KA=; b=KHPoaP5yfkNVlMEI18mCO5BF4BzqWf/1EgwarzMBvSDJtDOwmpYwxEDHAn3/eTFIJn tbsoiZ41x5O0aAY8VrCjiWYmWPxZFjeGDbldURMtg0G0oFXlHJ/kcfp1b22FMjO2+Wjs DGaoAxTo3Vn49cFDoCD7QfzAgRCSEDupi5K7Dw4ouMfq8yDcxf43KAfIM7/l76WeqUsI VbMGbP3LerQybGNimO97SlrOH91DT/o+Ea4gmNLNbS2sCCN9t4V/1LKEFIlISFBGi7Ga 4QGusGRS02oJOfilh2+O+H7TkXLuhyEjL4M1MVikoepGSa2o4wiIFLJZ23wFq+IWRGmi kjgg== X-Gm-Message-State: AOJu0YwJKAPBm+W0t7hZpF2x7oIyrpGLusHO59uCarKHtSl64gXOTmcs EGFNUZXebe/8XL5LAKAvwwem8mY9U6DtbxD/UUE/9iBj5MDbu+EgiLgUUr9evhP8IlPybJzImTr 0 X-Gm-Gg: ASbGncsmZkf6osllZWGMvqC0uKEzAVYSazJZDLuM/x2QaYEt4EpzhhHqhxa4mPGHrRZ MEt9uY+YEruFrhrVV2/PGBHdScIkUowK1C3CBtEMrUOJAMfxT85FdwIC2IqfBpT4k3BJ37c+QpU OzqMgsq0SP17wxrsB8JiTNDHBUVTSWZqZ1D8AbDOKiRJmqy1gFv/dNY9SoHe8qom/aAVXPAbzcr CFxDoNcaP3NKtiWLpRlBaAZBYcWy5XKTqmitkLm16pUfUcgSDRmqr7QZJ9AefOoqW4DhQswowyV 1GdhKurZcgy4UC+xUfcswQ== X-Google-Smtp-Source: AGHT+IGYuLIgynCbMW49zCAcXaCqe9PdpcO7rKOG6XcFvNDJEwBqe3T6NtQnpF2LUw82z7YgHNtv4A== X-Received: by 2002:a17:902:c408:b0:216:69ca:772a with SMTP id d9443c01a7336-219e6f297b3mr889970865ad.53.1736267135893; Tue, 07 Jan 2025 08:25:35 -0800 (PST) Received: from DougS18 (s66-183-142-209.bc.hsia.telus.net. [66.183.142.209]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dc96eb40sm312977135ad.86.2025.01.07.08.25.35 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 07 Jan 2025 08:25:35 -0800 (PST) From: "Doug Smythies" To: "'Peter Zijlstra'" Cc: , , "Doug Smythies" References: <005f01db5a44$3bb698e0$b323caa0$@telus.net> <20250106115732.GE20870@noisy.programming.kicks-ass.net> <000801db604b$e0f6b580$a2e42080$@telus.net> <20250106165932.GG20870@noisy.programming.kicks-ass.net> <20250106170455.GB22191@noisy.programming.kicks-ass.net> <001b01db608a$56d3dc40$047b94c0$@telus.net> <20250107112606.GN20870@noisy.programming.kicks-ass.net> <000d01db6115$69c1aef0$3d450cd0$@telus.net> In-Reply-To: <000d01db6115$69c1aef0$3d450cd0$@telus.net> Subject: RE: [REGRESSION] Re: [PATCH 00/24] Complete EEVDF Date: Tue, 7 Jan 2025 08:25:37 -0800 Message-ID: <001201db6120$c8eec2e0$5acc48a0$@telus.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 16.0 Content-Language: en-ca Thread-Index: AQJdvfVp9nqa9troPEySggPg+fzySgGTHC4vAVKdTGwBPBBbIAH1WJEKAgKs/R8BZUmAdQLzG1VksaMw8oA= On 2025.01.07 07:04 Doug Smythies wrote: > On 2025.01.07 03:26 Peter Zijlstra wrote: >> On Mon, Jan 06, 2025 at 02:28:40PM -0800, Doug Smythies wrote: >> If I get a blimp (>10000) then it is always on the last CPU, are you >> seeing the same thing? > > More or less, yes. The very long migrations are dominated by the > CPU 5 to CPU 11 migration. >> >>> In this short example all captures were for the CPU 5 to 11 migration. >>> 2 at 6 seconds, 1 at 1.33 seconds and 1 at 2 seconds. >> >> This seems to suggest you are, always on CPU 11. >> >> Weird! > > Yes, weird. I think, but am not certain, the CPU sequence in turbostat > per interval loop is: > > Wake on highest numbered CPU (11 in my case) > Do a bunch of work that can be done without MSR reads. > For each CPU in topological order (0,6,1,7,2,8,3,9,4,10,5,11 in my case) > Do the CPU specific work > Finish the intervals work and printing and such on CPU 11. > Sleep for the interval time (we have been using 1 second) > > Without any proof, I was thinking the CPU 11 dominance > for the long migration issue was due to the other bits of > work done on that CPU. To test this theory I hacked turbostat to migrate to CPU 3 After the CPU specific work loop. So now the per interval workflow is: Wake on CPU 3 Do a bunch of work that can be done without MSR reads. For each CPU in topological order (0,6,1,7,2,8,3,9,4,10,5,11 in my case) Do the CPU specific work Migrate to CPU 3 Finish the intervals work and printing and such on CPU 3. Sleep for the interval time And now I get: usec Time_Of_Day_Seconds CPU Busy% IRQ 12646 1736266361.533240 3 99.76 1005 6004653 1736266384.555240 3 99.76 1006 6004653 1736266393.563240 3 99.76 1004 6005648 1736266400.570240 3 99.76 7019 6005653 1736266432.602240 3 99.76 1005 6003656 1736266479.652242 3 99.76 1004 15636 1736266501.690240 3 99.76 1005 4948651 1736266528.661240 3 99.76 1004 521672 1736266534.192240 2 99.76 1002 1117651 1736266585.360239 3 99.76 1004 6003652 1736266592.365240 3 99.76 2123 3526648 1736266612.909240 3 99.76 1004 6003650 1736266632.927240 3 99.76 1005 396623 1736266636.327239 10 99.76 1002 6003654 1736266660.349240 3 99.76 1005 6003653 1736266682.369239 3 99.76 1006 6003653 1736266703.388240 3 99.76 1004 514673 1736266718.918240 2 99.76 1003 14652 1736266725.940240 3 99.76 1004 6003653 1736266745.958240 3 99.76 1004 6003653 1736266767.978240 3 99.76 1006 6003652 1736266794.002240 3 99.76 1006 6003653 1736266815.021240 3 99.76 1004 2496651 1736266841.542239 3 99.76 1007 6003647 1736266848.547240 3 99.76 3504 <<< 8 minutes 7 seconds elapsed