From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0F152C54EBE for ; Tue, 17 Jan 2023 01:46:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=2BxDAnMpDkH2YO5BcaugZI2D8ogzG5Grk0AI7zLR7CI=; b=kc1Zteo8EA5+ha fIWA52m50bY8VoqC+dxhoHR4fRjyINfAfrsNEXInKG9uGcX+8m1or5ECu5kkNoJuEu/MMLJTO5Qn0 hRCHcmmDSu95jdGdgeYLwgJ6uW0ypn071/Q2Bl/BHdg515RCgJUSmKGzstldWjiwgtg+1p5zT7JlM qA5wwz9XfEYfCJHc8q68kBGjRVZfglxNt6F5y8qNTOVJhl8t+pfUf3SxHoYX7ZwFoSxgMqqydDmcK QluYc1oTlQ8veegSGSmSJp5l0WFeEE+IFcNk1celCEbcXbgh/e+9Ygf6Ea6ebu79Zh4ixjxLsozqO hoZW0j8IaFNGDsw5ti9A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pHb2T-00CY2j-1R; Tue, 17 Jan 2023 01:45:25 +0000 Received: from mail-qt1-x831.google.com ([2607:f8b0:4864:20::831]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pHb2P-00CY0j-PL for linux-arm-kernel@lists.infradead.org; Tue, 17 Jan 2023 01:45:23 +0000 Received: by mail-qt1-x831.google.com with SMTP id h21so25980065qta.12 for ; Mon, 16 Jan 2023 17:45:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=dYhSlmk0PvxFX1+U3oYA9QkS2QMXHxwQiR9tSZnBUWw=; b=HyDb9cPhodKHbCX5PyjPWv3t75rpjuZ+D7bbvR5yYnKigDVr8kYw9TN2dMkilSHE/+ mU8scfWYBlgJrSR0ZfSmx9N9EPTbNk4nAG+Evh+CndALCfdINagMY6pHkGd6JdaH66i5 T9HMAlhIodPDY4HEx6Da+XhC1FiMZkSkglMc4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=dYhSlmk0PvxFX1+U3oYA9QkS2QMXHxwQiR9tSZnBUWw=; b=B2LxZS463zeYvIE4Q/sfS4mTS704kqsKClMjZJwzmFsAXe6GzIN9eXe0XIRGgjZQxs HcIO0PxnJTRiCOluuaHRTfdNF+7zEnO/I+tXLs/JfgigW4Fpwam2Rwvy9Duh7Zo62JI1 cITl1Nr490bB/PTNH2vvQ5q20Bixq8O3cnIQ4oSbRujNyAoU2EaHVpf6L8w8U3H4sHau v5ZsX/Ew1mvaT9OgM6TG1O11DVnBBVH+5Dj3Un8YWIC1++kHViaiHXFRyapB5ukhwtTW 36dE7rcbn3PuS7Dl5qEI8G3d8tmsNzQFzU5O9vt9Vd+AYo52U7PyrItxNzmXCXnXx8BJ zcbA== X-Gm-Message-State: AFqh2kqM4rNFIOaJPT+Ef5Zde31eTaacunRN9OF05b4xm8/3h8vPb2Dm 8g7MdB4sL3Kf6HCm77RZYW1LdA== X-Google-Smtp-Source: AMrXdXvz0tAieoyCSMNzPNvXGVUBJAnt7xFzPy8Wn+ZUkDoPr+m1Q7fpjxTUkX4N1uHafEQLvv+9Eg== X-Received: by 2002:ac8:4457:0:b0:3ab:5d1e:a775 with SMTP id m23-20020ac84457000000b003ab5d1ea775mr1178915qtn.12.1673919918020; Mon, 16 Jan 2023 17:45:18 -0800 (PST) Received: from localhost (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id x8-20020ac81208000000b003999d25e772sm15240106qti.71.2023.01.16.17.45.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Jan 2023 17:45:17 -0800 (PST) Date: Tue, 17 Jan 2023 01:45:16 +0000 From: Joel Fernandes To: Zhouyi Zhou Cc: "moderated list:ARM/STM32 ARCHITECTURE" , Will Deacon , Marc Zyngier , Mark Rutland , Catalin Marinas , rcu , "Paul E. McKenney" Subject: Re: arm64 torture test hotplug failures (offlining causes -EBUSY) Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230116_174521_852675_1C1EBC03 X-CRM114-Status: GOOD ( 39.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Jan 17, 2023 at 08:37:16AM +0800, Zhouyi Zhou wrote: > On Tue, Jan 17, 2023 at 8:15 AM Joel Fernandes wrote: > > > > On Mon, Jan 16, 2023 at 05:38:00PM -0500, Joel Fernandes wrote: > > > Hi Zhouyi, > > > > > > On Mon, Jan 16, 2023 at 1:33 PM Zhouyi Zhou wrote: > > > > > > > [..] > > > > On Tue, Jan 17, 2023 at 1:27 AM Joel Fernandes wrote: > > > > > > > > > > Hello, > > > > > I am seeing -EBUSY returned a lot during torture_onoff() when running > > > > > rcutorture on arm64. This causes hotplug failure 30% of the time. I am > > > > > also seeing this in 6.1-rc kernels. I believe see this only for CPU0. > > > > > > > > > > This causes warnings in torture tests: > > > > > [ 217.582290] rcu-torture:torture_onoff task: offline 0 failed: errno -16 > > > > > [ 221.866362] rcu-torture:torture_onoff task: offline 0 failed: errno -16 > > > > > > > > > > Full kernel log here: > > > > > http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TREE04/console.log > > > > > > > > > > Any ideas on why this is happening and only for CPU 0 (presumably the > > > > > boot CPU)? I'd personally need these warnings to go away for my tests > > > > > as this causes rcutorture's tests to not cleanly pass for me. It > > > > > appears remove_cpu() -> device_offline() is what returns the error. > > > > > > > > > I guess this probably because CPU 0 is the tick_do_timer_cpu in > > > > nohz_full mode, which prevent that cpu from > > > > going offline [1]. We have discussed this topic, but there is no > > > > agreement on how to solve it yet. > > > > > > But I am seeing the issue in TRACE02 config which is: > > > CONFIG_NO_HZ_IDLE=y > > > # CONFIG_NO_HZ_FULL is not set > > > > > > So that is not NO_HZ_FULL: > > > http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/console.log.diags/ > > > However, I can't seem to find the full kernel logs for that. > > > > > > Also, other than the TRACE02 fail, I only see the issue with configs > > > with CONFIG_NO_HZ_FULL=y > > > > > > Can you try TRACE02 specifically, and see if you can reproduce the > > > same issue on your setup? Meanwhile, I'll try to trace what is > > > returning the -EBUSY. > I am trying TRACE02 on my X86_64 machine using cross compile and > qemu-system-aarch64 now, my equipment is limited, but hope I can be of > beneficial to the community ;-) Cool, I am assuming you are trying the patch you shared which you wrote in November. I bet you will still see the issue. > > > > How about something simple like the following? (untested) > > > > ---8<----------------------- > > > > diff --git a/kernel/torture.c b/kernel/torture.c > > index bc8fb361efc0..cd64110694c0 100644 > > --- a/kernel/torture.c > > +++ b/kernel/torture.c > > @@ -220,6 +220,9 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes, > > // PCI probe frequently disables hotplug during boot. > > (*n_offl_attempts)--; > > s = " (-EBUSY forgiven during boot)"; > > + } else if (tick_nohz_full_running && ret == -EBUSY) { > > + (*n_offl_attempts)--; > > + s = " (-EBUSY forgiven if nohz_full is running)"; > Fantastic fix!! thus we can fix the time keeper cpu torture problem > without touch the time keeper code. Thanks. Unfortunately this does not fix the issue for TRACE02 and the patch you shared does not fix it either -- because TRACE02 is not a no-hz-full test. :-( We will need to do a bit of tracing to figure out where the -EBUSY is coming from for TRACE02. I wonder if we should ignore -EBUSY altogether, since as Thomas mentioned, hotplug failure is "normal". Thoughts? thanks, - Joel _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel