From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CFE067F7F2; Wed, 24 Jan 2024 17:41:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.180 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706118082; cv=none; b=Uk7OIwcgdGmO6fcVgsHAPrEFoAjCmkQioAqvH+93o7Fh6JsWrkeeOrZOIMbS5Pe/5dwf4PgOnobB7r/kO5anncA4FjL0ODIGRczFEHrY7C9imejCrW7MihtZKHYm2QA9wEszYT7RRC6KKZBBhZfdDeL9mM5IUlxN/WeBrwZ/03k= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706118082; c=relaxed/simple; bh=tw/ivyCBJnZU8g06Ui6BqgkgwG6NCYQYJWdwrEbM3kk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=cSjtuzbZNbogPLgKRE0GjDpMZXi+bVag+INxfIeLFayqGzsE15FV5xX9LgonPjhxKBvHg24z9n9e1HNIj8PLfep8QUnT9ABJL3+NNYQkg3eoZQDvV7OrSAey113JwLzMFwhVm6jtEjb7DOTWWHhasgpuKjTyGYTc5OeFI8XEAWM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=G5EdBFOr; arc=none smtp.client-ip=209.85.128.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="G5EdBFOr" Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-5ff84214fc7so56504587b3.0; Wed, 24 Jan 2024 09:41:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706118080; x=1706722880; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=HrKSWjAPLury0l0O07YwABLahdqYSfqu/vkyFWYooQ8=; b=G5EdBFOrAMr19wXn2q14X3LktuaZwjb1JaH9G+/9xLGwXHmGz3H/Bg/o9H/g+d5zTd MB7isfnGoAtjLDqpLhEPl48t1p0VST3rziSB78N+5rZfq915mUvuC3e2Z0cHLw8EkLe6 7CmdbkRX9Ximcvf6CuJZLPPjugNw5kzYzMV1VUNRFXR7euEYBl9dIojpDZVjijgPXXer b2peL5QCepqqnrHucuRUg9L8FTjSx03nSx3g8WQjBpHFosE9urZenIJ2MXfU/LvijYvQ fFp+LtHhW0/Mn8aVc+YlF+exhgK2fQCH0NIh+lOJgSaP4ZP1aN3DqsexWp9RDkyAGgY5 /0LA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706118080; x=1706722880; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=HrKSWjAPLury0l0O07YwABLahdqYSfqu/vkyFWYooQ8=; b=MDhAxeb3LoGADwktNboej6gHeMACe96qXGog3FeumAwMpNzKxNEQpV2i4D4SuXerNW As6CYk484OsGwejYoQVcU99DPlxhkSGoLAi99WiNGyQr1fuKE9KcS6mPNRfhxlkiohph eV9Priu+YLfX39cRcaguUaXKT0G7kIFA2NM1v+UG32LebNIMSjAWqZG86dQoOxVW0Bzq SdCS5PgY7BONS01CpME/hq5XuHHvjW3HUvNoNuZJf6rDp2fF7xul7ZA+2VHojw+oG9Kk +kcYksoXY5WHHnjaZYtdq67wtDRvK5hgew/M1wckILDiupu8DH5dinQ7tkvIoj9BrGCx AQMg== X-Gm-Message-State: AOJu0YyHdTSImhU/F5BF606d1InoH42aedxQTHfmFSJC3q+Sf/oYyrTB YfsAmX6VbBTgkIWCNjW7azF3Nm7jCuAaJIwDMwEJ3BuuIajmUg7t X-Google-Smtp-Source: AGHT+IG8FJ68JYgvol2vsYmkZB9x7Cw6vyk82+V1soudJ0xN7Bq1azDGVFF6zVhVo1ZMyf65R6wE/A== X-Received: by 2002:a0d:d70b:0:b0:5ff:9f2c:1299 with SMTP id z11-20020a0dd70b000000b005ff9f2c1299mr1077412ywd.65.1706118079707; Wed, 24 Jan 2024 09:41:19 -0800 (PST) Received: from localhost ([2601:344:8301:57f0:abdb:7236:6977:9ab5]) by smtp.gmail.com with ESMTPSA id t140-20020a0dea92000000b005ffb2815960sm77564ywe.45.2024.01.24.09.41.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jan 2024 09:41:19 -0800 (PST) Date: Wed, 24 Jan 2024 09:41:18 -0800 From: Yury Norov To: "Lameter, Christopher" Cc: Huang Shijie , gregkh@linuxfoundation.org, patches@amperecomputing.com, rafael@kernel.org, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, kuba@kernel.org, vschneid@redhat.com, mingo@kernel.org, akpm@linux-foundation.org, vbabka@suse.cz, rppt@kernel.org, tglx@linutronix.de, jpoimboe@kernel.org, ndesaulniers@google.com, mikelley@microsoft.com, mhiramat@kernel.org, arnd@arndb.de, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, mark.rutland@arm.com, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, chenhuacai@kernel.org, jiaxun.yang@flygoat.com, linux-mips@vger.kernel.org Subject: Re: [PATCH v2] NUMA: Early use of cpu_to_node() returns 0 instead of the correct node id Message-ID: References: <20240123045843.75969-1-shijie@os.amperecomputing.com> <4a13353c-cf4b-a388-5776-389c61c63ec0@os.amperecomputing.com> Precedence: bulk X-Mailing-List: linux-mips@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4a13353c-cf4b-a388-5776-389c61c63ec0@os.amperecomputing.com> On Wed, Jan 24, 2024 at 09:19:00AM -0800, Lameter, Christopher wrote: > On Tue, 23 Jan 2024, Huang Shijie wrote: > > > During the kernel booting, the generic cpu_to_node() is called too early in > > arm64, powerpc and riscv when CONFIG_NUMA is enabled. > > > > For arm64/powerpc/riscv, there are at least four places in the common code > > where the generic cpu_to_node() is called before it is initialized: > > 1.) early_trace_init() in kernel/trace/trace.c > > 2.) sched_init() in kernel/sched/core.c > > 3.) init_sched_fair_class() in kernel/sched/fair.c > > 4.) workqueue_init_early() in kernel/workqueue.c > > > > In order to fix the bug, the patch changes generic cpu_to_node to > > function pointer, and export it for kernel modules. > > Introduce smp_prepare_boot_cpu_start() to wrap the original > > smp_prepare_boot_cpu(), and set cpu_to_node with early_cpu_to_node. > > Introduce smp_prepare_cpus_done() to wrap the original smp_prepare_cpus(), > > and set the cpu_to_node to formal _cpu_to_node(). > > Would you please fix this cleanly without a function pointer? > > What I think needs to be done is a patch series. > > 1. Instrument cpu_to_node so that some warning is issued if it is used too > early. Preloading the array with NUMA_NO_NODE would allow us to do that. By preloading do you mean compile-time initialization? > 2. Implement early_cpu_to_node on platforms that currently do not have it. > > 3. A series of patches that fix each place where cpu_to_node is used too > early. Agree. This is the right way to go. And pretty well all of it was discussed in v1, isn't? Thanks, Yury From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D2200C47E49 for ; Wed, 24 Jan 2024 17:41:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ZaPlBg2gOi0BFlzSQa1N7G2DiXEn/0R/c6ixqcyXIi8=; b=Ot9jZv0i57h5zM +dDwOqhXLmD0wKBzeXI8oDFpWT1j1iJxWFfvwzlbgZaII24dWaqIuhkxQTQxLA88P/AN1ssebitBK ED040wEkY1vnkk8SEs0gCqk175tBzEeraT+YKe3EX20gwOz1wlQqgqkSdogKE+jvhwpS1gQQlvrYS MPyrBR0UIKbSO89k3DQepbQFWPMqzFz3KjmIGGzuEncKPAKA0FYIsRpQIgB00ountPblB015FAoNp P5+98TqTlFvWs/lWSUXK6VtW8ASTpOmkzDxuFvKmGnHUsfk9S27/XF78fNQKD6/0HS8texSseN495 jpnwe8e9PxWeoWycJZSg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rShFc-004Xjb-1g; Wed, 24 Jan 2024 17:41:24 +0000 Received: from mail-yw1-x1136.google.com ([2607:f8b0:4864:20::1136]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rShFZ-004XiE-2H; Wed, 24 Jan 2024 17:41:23 +0000 Received: by mail-yw1-x1136.google.com with SMTP id 00721157ae682-5ffe7e7b7b3so35083647b3.3; Wed, 24 Jan 2024 09:41:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706118080; x=1706722880; darn=lists.infradead.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=HrKSWjAPLury0l0O07YwABLahdqYSfqu/vkyFWYooQ8=; b=QtvDdF6NbEp67GtnOb1eHAGDtrCkUCiIx0tJwG0zcWGNiq8eAPC1vz6rwWTk0Xkyt/ K34/eaIj4PaV0sVVIAM4sHQYwRdZZlpD/Tdu3tAbVDAQofD7hoEoSmPXxNGlPT1iEs2c /h7LEC0cq6BVv/G/IuCbjpbdHlTH2xFf1U0EP8NIjrnIXumeoWIxT3zE9BuYnwhR3iu8 wnRItB8McOUp6fc868J2Chlu2AVGlAJWwkZ+RQjYEC9UN40Hk1TfJCY+XYHo5R9S/DGC jTAJAtT+ITFybzMB6Vimi1x7dQ7TAUOjrZxoecFoN8aJnP5+kzfAR+kzPae2C+Hkatom vNhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706118080; x=1706722880; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=HrKSWjAPLury0l0O07YwABLahdqYSfqu/vkyFWYooQ8=; b=DB0TVe5LnR/fg83wQmZ9ZMh4zngnkPzzV2SeVQAapJwWMjSvofd05JANNx/wx81AyX ZYH3l9YnuT8ofW48Hn+c72tkokoq6X1yRZS0cZsPZW6P11lQyY0F5YAONPJgPXvvchYR U0pIKRaTMJzA7eumDU0nlU2LbJh9MhZGHikyTkHQoq5q+4A1PsTRMSo0JwTAEmETgh9D irBi6xrY/OFFE72HyQOfsz99TETlhofHkz4xGDAt3dgOOgxEGpQFB3AjhuHJmrHlum/T WMoM/KCLRCMBNw+fKBUv3gUBi/J3M45UxDxHfH+qN+UASsARbzb5dEj43yWQm6znuZ+5 IylA== X-Gm-Message-State: AOJu0Yzna7W9bT5pigVSxqL4E/DBvzMmyBYHr0fokTMHO5PH5tYRY3g3 /lz9l8QBEw4SjW7hZj5KuVYeBOgE3OwK+FIl0Fv3FBade21r/Zc8IeG0+uSYSl0= X-Google-Smtp-Source: AGHT+IG8FJ68JYgvol2vsYmkZB9x7Cw6vyk82+V1soudJ0xN7Bq1azDGVFF6zVhVo1ZMyf65R6wE/A== X-Received: by 2002:a0d:d70b:0:b0:5ff:9f2c:1299 with SMTP id z11-20020a0dd70b000000b005ff9f2c1299mr1077412ywd.65.1706118079707; Wed, 24 Jan 2024 09:41:19 -0800 (PST) Received: from localhost ([2601:344:8301:57f0:abdb:7236:6977:9ab5]) by smtp.gmail.com with ESMTPSA id t140-20020a0dea92000000b005ffb2815960sm77564ywe.45.2024.01.24.09.41.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jan 2024 09:41:19 -0800 (PST) Date: Wed, 24 Jan 2024 09:41:18 -0800 From: Yury Norov To: "Lameter, Christopher" Cc: Huang Shijie , gregkh@linuxfoundation.org, patches@amperecomputing.com, rafael@kernel.org, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, kuba@kernel.org, vschneid@redhat.com, mingo@kernel.org, akpm@linux-foundation.org, vbabka@suse.cz, rppt@kernel.org, tglx@linutronix.de, jpoimboe@kernel.org, ndesaulniers@google.com, mikelley@microsoft.com, mhiramat@kernel.org, arnd@arndb.de, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, mark.rutland@arm.com, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, chenhuacai@kernel.org, jiaxun.yang@flygoat.com, linux-mips@vger.kernel.org Subject: Re: [PATCH v2] NUMA: Early use of cpu_to_node() returns 0 instead of the correct node id Message-ID: References: <20240123045843.75969-1-shijie@os.amperecomputing.com> <4a13353c-cf4b-a388-5776-389c61c63ec0@os.amperecomputing.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <4a13353c-cf4b-a388-5776-389c61c63ec0@os.amperecomputing.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240124_094121_742843_FB7E3977 X-CRM114-Status: GOOD ( 22.57 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On Wed, Jan 24, 2024 at 09:19:00AM -0800, Lameter, Christopher wrote: > On Tue, 23 Jan 2024, Huang Shijie wrote: > > > During the kernel booting, the generic cpu_to_node() is called too early in > > arm64, powerpc and riscv when CONFIG_NUMA is enabled. > > > > For arm64/powerpc/riscv, there are at least four places in the common code > > where the generic cpu_to_node() is called before it is initialized: > > 1.) early_trace_init() in kernel/trace/trace.c > > 2.) sched_init() in kernel/sched/core.c > > 3.) init_sched_fair_class() in kernel/sched/fair.c > > 4.) workqueue_init_early() in kernel/workqueue.c > > > > In order to fix the bug, the patch changes generic cpu_to_node to > > function pointer, and export it for kernel modules. > > Introduce smp_prepare_boot_cpu_start() to wrap the original > > smp_prepare_boot_cpu(), and set cpu_to_node with early_cpu_to_node. > > Introduce smp_prepare_cpus_done() to wrap the original smp_prepare_cpus(), > > and set the cpu_to_node to formal _cpu_to_node(). > > Would you please fix this cleanly without a function pointer? > > What I think needs to be done is a patch series. > > 1. Instrument cpu_to_node so that some warning is issued if it is used too > early. Preloading the array with NUMA_NO_NODE would allow us to do that. By preloading do you mean compile-time initialization? > 2. Implement early_cpu_to_node on platforms that currently do not have it. > > 3. A series of patches that fix each place where cpu_to_node is used too > early. Agree. This is the right way to go. And pretty well all of it was discussed in v1, isn't? Thanks, Yury _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F3CAEC46CD2 for ; Wed, 24 Jan 2024 17:42:15 +0000 (UTC) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Ym3oj3aL; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4TKrr62Vrgz3cCS for ; Thu, 25 Jan 2024 04:42:14 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Ym3oj3aL; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::1130; helo=mail-yw1-x1130.google.com; envelope-from=yury.norov@gmail.com; receiver=lists.ozlabs.org) Received: from mail-yw1-x1130.google.com (mail-yw1-x1130.google.com [IPv6:2607:f8b0:4864:20::1130]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4TKrqB1PJYz3btl for ; Thu, 25 Jan 2024 04:41:25 +1100 (AEDT) Received: by mail-yw1-x1130.google.com with SMTP id 00721157ae682-5f254d1a6daso60002057b3.2 for ; Wed, 24 Jan 2024 09:41:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706118080; x=1706722880; darn=lists.ozlabs.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=HrKSWjAPLury0l0O07YwABLahdqYSfqu/vkyFWYooQ8=; b=Ym3oj3aLehX5s/uyltabOGLXGs/sSeBuhe2awRHExIYPIazPLtQLrw1jRMlmBKRwu/ LNQ0G0gSTgLKYoxL2u/+994EfqqUw9QkdkKN2igQi8F5spf1ShV9I87LdA6rz+R85ZBS gws5bmZGBal23z4Q7RdaOi53SrAkJSBZDDOlExBYcrQdG4pRlEhXFpnadmFILtTbChub +m9jEYSpXurAlyvKVr2cGsW9capB2vBVfVTDV1etg2xFiUKRIQecRfZr/zLnCKiEQwMx vc/lM7pPMMe1qahrSF4z1YYigY8D4P3bV4oJWsa1Xou5jdpvhwb8ddN/zW3X7Z/FsLSh EiLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706118080; x=1706722880; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=HrKSWjAPLury0l0O07YwABLahdqYSfqu/vkyFWYooQ8=; b=X21iZWBTk3U70MpRD3OfFNv+0AUShhD3SC/bR72CeTIpTv6nEhCyZko52Is1040/Gw R8+9aI8RCEiKNask1y8ny37vz8l3uX/IszgX1bM1TUguEkVXbncW1Q6eKTRCQ/WWC7yz f/hkifL4vxE8DVylmoHcHndR2T0zTfBx39Umz9lpOsXT2qG83Q3JeHHtW22W4mvGsiAM Snpfo1Zju2aeB13GnYFXEORz2xnn7Pv1jzYZ7SEMH+e9a9UngAhRcLdH6TJ/2G4faixY h6XRtVEe1WS9vyJhG40ysuOyCgnz76NdsY/xMF1tBdNfKnmXAX01FHzdaXTn93m2RMBi WK1w== X-Gm-Message-State: AOJu0Yx6wAhHiTeWqv5dulic52SCyQT6ij49ua0jzcOw8kMYoL08ahz/ orb+WS+YF71scOpoNXhRb/w6APxHrEZ2ieD0Zf6HIjG2vdMladae X-Google-Smtp-Source: AGHT+IG8FJ68JYgvol2vsYmkZB9x7Cw6vyk82+V1soudJ0xN7Bq1azDGVFF6zVhVo1ZMyf65R6wE/A== X-Received: by 2002:a0d:d70b:0:b0:5ff:9f2c:1299 with SMTP id z11-20020a0dd70b000000b005ff9f2c1299mr1077412ywd.65.1706118079707; Wed, 24 Jan 2024 09:41:19 -0800 (PST) Received: from localhost ([2601:344:8301:57f0:abdb:7236:6977:9ab5]) by smtp.gmail.com with ESMTPSA id t140-20020a0dea92000000b005ffb2815960sm77564ywe.45.2024.01.24.09.41.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jan 2024 09:41:19 -0800 (PST) Date: Wed, 24 Jan 2024 09:41:18 -0800 From: Yury Norov To: "Lameter, Christopher" Subject: Re: [PATCH v2] NUMA: Early use of cpu_to_node() returns 0 instead of the correct node id Message-ID: References: <20240123045843.75969-1-shijie@os.amperecomputing.com> <4a13353c-cf4b-a388-5776-389c61c63ec0@os.amperecomputing.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4a13353c-cf4b-a388-5776-389c61c63ec0@os.amperecomputing.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, rafael@kernel.org, catalin.marinas@arm.com, jiaxun.yang@flygoat.com, mikelley@microsoft.com, linux-riscv@lists.infradead.org, will@kernel.org, mingo@kernel.org, vschneid@redhat.com, chenhuacai@kernel.org, vbabka@suse.cz, kuba@kernel.org, patches@amperecomputing.com, linux-mips@vger.kernel.org, aou@eecs.berkeley.edu, arnd@arndb.de, paul.walmsley@sifive.com, tglx@linutronix.de, jpoimboe@kernel.org, linux-arm-kernel@lists.infradead.org, Huang Shijie , gregkh@linuxfoundation.org, ndesaulniers@google.com, linux-kernel@vger.kernel.org, palmer@dabbelt.com, mhiramat@kernel.org, akpm@linux-foundation.org, linuxppc-dev@lists.ozlabs.org, rppt@kernel.org Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Wed, Jan 24, 2024 at 09:19:00AM -0800, Lameter, Christopher wrote: > On Tue, 23 Jan 2024, Huang Shijie wrote: > > > During the kernel booting, the generic cpu_to_node() is called too early in > > arm64, powerpc and riscv when CONFIG_NUMA is enabled. > > > > For arm64/powerpc/riscv, there are at least four places in the common code > > where the generic cpu_to_node() is called before it is initialized: > > 1.) early_trace_init() in kernel/trace/trace.c > > 2.) sched_init() in kernel/sched/core.c > > 3.) init_sched_fair_class() in kernel/sched/fair.c > > 4.) workqueue_init_early() in kernel/workqueue.c > > > > In order to fix the bug, the patch changes generic cpu_to_node to > > function pointer, and export it for kernel modules. > > Introduce smp_prepare_boot_cpu_start() to wrap the original > > smp_prepare_boot_cpu(), and set cpu_to_node with early_cpu_to_node. > > Introduce smp_prepare_cpus_done() to wrap the original smp_prepare_cpus(), > > and set the cpu_to_node to formal _cpu_to_node(). > > Would you please fix this cleanly without a function pointer? > > What I think needs to be done is a patch series. > > 1. Instrument cpu_to_node so that some warning is issued if it is used too > early. Preloading the array with NUMA_NO_NODE would allow us to do that. By preloading do you mean compile-time initialization? > 2. Implement early_cpu_to_node on platforms that currently do not have it. > > 3. A series of patches that fix each place where cpu_to_node is used too > early. Agree. This is the right way to go. And pretty well all of it was discussed in v1, isn't? Thanks, Yury From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 25A0BC46CD2 for ; Wed, 24 Jan 2024 17:41:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IQYUCSpV5qW0mtE02WXU7qYtlubK3Ovh4leq7Zm+PvA=; b=RUW6R8ICs/u2uv OkP4cW7SKbavYB1xaRyKAXrbyyy5TKtV70QCNXr68r1LhXbH21zNK51qVNrxC20XWdDdLK8fh/hqy mhQmjwrVWh/Vljx5MC6DxcJSswdvJO3GA9lH76+eKgeZh2c8kPjLESQU3+F0AliZkI0JRNRG/v2G/ DfIFP+W6xL2VYcWkF/CiyhjqwB9WeA2Z4rIr3Jlg5JzPUvRXJtgUEpyyiekNXJA73jUJ7NlHHb0i7 Xkp6jkh4XONtOUz/9iTzSoOsv/AMWyhjQcZ8MkfUeB6OD+xaCt4idn8zZpCpX3jqvL9RnCQ5Ubelf 2xkxSorsCGhGB4HNtzCw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rShFd-004Xjr-0s; Wed, 24 Jan 2024 17:41:25 +0000 Received: from mail-yw1-x1136.google.com ([2607:f8b0:4864:20::1136]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rShFZ-004XiE-2H; Wed, 24 Jan 2024 17:41:23 +0000 Received: by mail-yw1-x1136.google.com with SMTP id 00721157ae682-5ffe7e7b7b3so35083647b3.3; Wed, 24 Jan 2024 09:41:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706118080; x=1706722880; darn=lists.infradead.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=HrKSWjAPLury0l0O07YwABLahdqYSfqu/vkyFWYooQ8=; b=QtvDdF6NbEp67GtnOb1eHAGDtrCkUCiIx0tJwG0zcWGNiq8eAPC1vz6rwWTk0Xkyt/ K34/eaIj4PaV0sVVIAM4sHQYwRdZZlpD/Tdu3tAbVDAQofD7hoEoSmPXxNGlPT1iEs2c /h7LEC0cq6BVv/G/IuCbjpbdHlTH2xFf1U0EP8NIjrnIXumeoWIxT3zE9BuYnwhR3iu8 wnRItB8McOUp6fc868J2Chlu2AVGlAJWwkZ+RQjYEC9UN40Hk1TfJCY+XYHo5R9S/DGC jTAJAtT+ITFybzMB6Vimi1x7dQ7TAUOjrZxoecFoN8aJnP5+kzfAR+kzPae2C+Hkatom vNhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706118080; x=1706722880; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=HrKSWjAPLury0l0O07YwABLahdqYSfqu/vkyFWYooQ8=; b=DB0TVe5LnR/fg83wQmZ9ZMh4zngnkPzzV2SeVQAapJwWMjSvofd05JANNx/wx81AyX ZYH3l9YnuT8ofW48Hn+c72tkokoq6X1yRZS0cZsPZW6P11lQyY0F5YAONPJgPXvvchYR U0pIKRaTMJzA7eumDU0nlU2LbJh9MhZGHikyTkHQoq5q+4A1PsTRMSo0JwTAEmETgh9D irBi6xrY/OFFE72HyQOfsz99TETlhofHkz4xGDAt3dgOOgxEGpQFB3AjhuHJmrHlum/T WMoM/KCLRCMBNw+fKBUv3gUBi/J3M45UxDxHfH+qN+UASsARbzb5dEj43yWQm6znuZ+5 IylA== X-Gm-Message-State: AOJu0Yzna7W9bT5pigVSxqL4E/DBvzMmyBYHr0fokTMHO5PH5tYRY3g3 /lz9l8QBEw4SjW7hZj5KuVYeBOgE3OwK+FIl0Fv3FBade21r/Zc8IeG0+uSYSl0= X-Google-Smtp-Source: AGHT+IG8FJ68JYgvol2vsYmkZB9x7Cw6vyk82+V1soudJ0xN7Bq1azDGVFF6zVhVo1ZMyf65R6wE/A== X-Received: by 2002:a0d:d70b:0:b0:5ff:9f2c:1299 with SMTP id z11-20020a0dd70b000000b005ff9f2c1299mr1077412ywd.65.1706118079707; Wed, 24 Jan 2024 09:41:19 -0800 (PST) Received: from localhost ([2601:344:8301:57f0:abdb:7236:6977:9ab5]) by smtp.gmail.com with ESMTPSA id t140-20020a0dea92000000b005ffb2815960sm77564ywe.45.2024.01.24.09.41.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jan 2024 09:41:19 -0800 (PST) Date: Wed, 24 Jan 2024 09:41:18 -0800 From: Yury Norov To: "Lameter, Christopher" Cc: Huang Shijie , gregkh@linuxfoundation.org, patches@amperecomputing.com, rafael@kernel.org, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, kuba@kernel.org, vschneid@redhat.com, mingo@kernel.org, akpm@linux-foundation.org, vbabka@suse.cz, rppt@kernel.org, tglx@linutronix.de, jpoimboe@kernel.org, ndesaulniers@google.com, mikelley@microsoft.com, mhiramat@kernel.org, arnd@arndb.de, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, mark.rutland@arm.com, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, chenhuacai@kernel.org, jiaxun.yang@flygoat.com, linux-mips@vger.kernel.org Subject: Re: [PATCH v2] NUMA: Early use of cpu_to_node() returns 0 instead of the correct node id Message-ID: References: <20240123045843.75969-1-shijie@os.amperecomputing.com> <4a13353c-cf4b-a388-5776-389c61c63ec0@os.amperecomputing.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <4a13353c-cf4b-a388-5776-389c61c63ec0@os.amperecomputing.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240124_094121_742843_FB7E3977 X-CRM114-Status: GOOD ( 22.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Jan 24, 2024 at 09:19:00AM -0800, Lameter, Christopher wrote: > On Tue, 23 Jan 2024, Huang Shijie wrote: > > > During the kernel booting, the generic cpu_to_node() is called too early in > > arm64, powerpc and riscv when CONFIG_NUMA is enabled. > > > > For arm64/powerpc/riscv, there are at least four places in the common code > > where the generic cpu_to_node() is called before it is initialized: > > 1.) early_trace_init() in kernel/trace/trace.c > > 2.) sched_init() in kernel/sched/core.c > > 3.) init_sched_fair_class() in kernel/sched/fair.c > > 4.) workqueue_init_early() in kernel/workqueue.c > > > > In order to fix the bug, the patch changes generic cpu_to_node to > > function pointer, and export it for kernel modules. > > Introduce smp_prepare_boot_cpu_start() to wrap the original > > smp_prepare_boot_cpu(), and set cpu_to_node with early_cpu_to_node. > > Introduce smp_prepare_cpus_done() to wrap the original smp_prepare_cpus(), > > and set the cpu_to_node to formal _cpu_to_node(). > > Would you please fix this cleanly without a function pointer? > > What I think needs to be done is a patch series. > > 1. Instrument cpu_to_node so that some warning is issued if it is used too > early. Preloading the array with NUMA_NO_NODE would allow us to do that. By preloading do you mean compile-time initialization? > 2. Implement early_cpu_to_node on platforms that currently do not have it. > > 3. A series of patches that fix each place where cpu_to_node is used too > early. Agree. This is the right way to go. And pretty well all of it was discussed in v1, isn't? Thanks, Yury _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel