From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KmipKZQa" Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0D3910C0; Wed, 29 Nov 2023 13:20:45 -0800 (PST) Received: by mail-ej1-x635.google.com with SMTP id a640c23a62f3a-a132acb67b5so30308766b.3; Wed, 29 Nov 2023 13:20:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701292844; x=1701897644; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=e7uJ6/cKIhORoXu8TzZSGHhq752ZWeeIg37a8KHIARo=; b=KmipKZQaD7eBdQGjrHhsDnIdiispL672PDDG3mL3WJMSUGRsgdIfnA4f0SDvU8/zUg DwfSf1QmxWoNZvv4Njtvk4KcMq4hMNBYIaFQBkcClDVdnl798K0pz91VnZ8CkdFZC/Pa VN4JM7Se4CIqpf5k1pZHbuNOjWGHg2Ae6q35Pc1eBYZEkD1SSFQivPMCazTFSiE5YEcv nabHWacSSM0Jk0/mRzM82TCToaRRoyyA2kSLV22GILbspPzfYM07LQ5KXWi059/Ru08l wEyBVILxXjZVZZMp2Sw27iyGsCx9gCDM2DYpD3iwaqXAaNJRtVNWqIDf2Pgljk3rh48w 9zSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701292844; x=1701897644; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=e7uJ6/cKIhORoXu8TzZSGHhq752ZWeeIg37a8KHIARo=; b=eJIdWYP5tWk0e8dnujJ6ASbDRZDngG9bO5gUMML/LpNthe6OlhEYg2mvhRRBjyCNfD 0rU6jzV1XhvG3BI26o72BwnqjBtZyfFTqA3ZJ+uwowrzSfHV5HCwV86sQX+exvrM5t16 Q4yJ+cyVjdAyTOrN+Kj7CsQU2fJTD0cDCYQmiVyFfyTIRhSWzh0b2Mx9WHB9KlP6ClvN gtH4igoUB+x9e1+mMR/rzhADdaCUOJzxzEmz5T1w2HocCfZ5rkG50REJkvuDZOa80z9l oVW6DmHwqf6lwvu40EKScm0+XvC33Di0vjJSe5iyUq27k/5pYuGwRLYOuWijF6h4hFam 8gRQ== X-Gm-Message-State: AOJu0Yzrfl8JrDEMn+YXwGM4e1F1Nrrm9bMbCHaT6NAERUPYz4uh/Sgj YwJzUXEnaPoHKWX0+5f6vl4= X-Google-Smtp-Source: AGHT+IHW256ZYrusTlNvNtiLRMCdgMj6p64wZO364+KYmDxL9yXjM8VrqXHYyIuRwwYq8Ld2nVxSCA== X-Received: by 2002:a17:906:27d9:b0:a01:c04a:ca9b with SMTP id k25-20020a17090627d900b00a01c04aca9bmr14609716ejc.72.1701292843807; Wed, 29 Nov 2023 13:20:43 -0800 (PST) Received: from [192.168.26.149] (031011218106.poznan.vectranet.pl. [31.11.218.106]) by smtp.googlemail.com with ESMTPSA id p18-20020a1709061b5200b009b9a1714524sm8409276ejg.12.2023.11.29.13.20.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Nov 2023 13:20:41 -0800 (PST) Message-ID: Date: Wed, 29 Nov 2023 22:20:38 +0100 Precedence: bulk X-Mailing-List: linux-clk@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: ARM BCM53573 SoC hangs/lockups caused by locks/clock/random changes To: Linus Walleij Cc: Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Russell King , Daniel Lezcano , Thomas Gleixner , Florian Fainelli , linux-clk@vger.kernel.org, linux-arm-kernel@lists.infradead.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, openwrt-devel@lists.openwrt.org, bcm-kernel-feedback-list@broadcom.com References: Content-Language: en-US From: =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi, it's a late reply but I didn't find enough determination earlier. On 8.09.2023 10:10, Linus Walleij wrote: > On Mon, Sep 4, 2023 at 10:34 AM Rafał Miłecki wrote: > >> I'm clueless at this point. >> Maybe someone can come up with an idea of actual issue & ideally a >> solution. > > Damn this is frustrating. > >> 2. Clock (arm,armv7-timer) >> >> While comparing main clock in Broadcom's SDK with upstream one I noticed >> a tiny difference: mask value. I don't know it it makes any sense but >> switching from CLOCKSOURCE_MASK(56) to CLOCKSOURCE_MASK(64) in >> arm_arch_timer.c (to match SDK) increases average uptime (time before a >> hang/lockup happens) from 4 minutes to 36 minutes. > > This could be related to how often the system goes to idle. > >> + if (cpu_idle_force_poll == 1234) >> + arch_cpu_idle(); >> + if (cpu_idle_force_poll == 5678) >> + arch_cpu_idle(); >> + if (cpu_idle_force_poll == 1234) >> + arch_cpu_idle(); >> + if (cpu_idle_force_poll == 5678) >> + arch_cpu_idle(); >> + if (cpu_idle_force_poll == 1234) >> + arch_cpu_idle(); >> + if (cpu_idle_force_poll == 5678) >> + arch_cpu_idle(); >> + if (cpu_idle_force_poll == 1234) >> + arch_cpu_idle(); > > Idle again. > > I would have tried to see what arch_cpu_idle() is doing. > > arm_pm_idle() or cpu_do_idle()? In my case arm_pm_idle is NULL. > What happens if you just put return in arch_cpu_idle() > so it does nothing? Doesn't help. I also tried putting: udelay(10); and udelay(1000); at the arch_cpu_idle() beginning. None helped. Here comes more interesting experiment though. Putting there: if (!(foo++ % 10000)) { pr_info("[%s] arm_pm_idle:%ps\n", __func__, arm_pm_idle); } doesn't seem to help. Putting following however seems to make kernel/device stable: if (!(foo++ % 100)) { pr_info("[%s] arm_pm_idle:%ps\n", __func__, arm_pm_idle); } I think I'm just going to assume those chipsets are simply hw broken. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A40BBC4167B for ; Wed, 29 Nov 2023 21:21:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To:Subject: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=FOpM1QGNKSNf/hwQ5UcBp0ZGsWFxL2JBaIlp6qCmgyY=; b=qxVobvvyh05DL7 a1h032qqUfJgBTWTqDCKGEMzwN4Tjj8f7BhWs2o7cEMyKUiayCp4XAJYuubEkXJT7xYq6K3TQIYAD VKx2mmU2nGNbesCnEfuH8I918BZAmmQs4qQiKvDzBiiEA9a4bal0LuI2L0HpJ0DBx9XuQ5SPjamw+ f/0iRPN+BiGQKNsgwI/EqDaPxyXmxJGtVdu1R3XzWHeQRaEYsLgz148KGww55eqZXz5FkPOYLJ2JQ CLshDfdacdRZYfenZUS6kMJzKHNI7IdXx+Os/sdgDXmV4oF9KyuP3NCk0uVY1DaM1Q5QKmlcXxE2a gfHYeUMddJQYmMTinoeQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r8RzH-009JFM-24; Wed, 29 Nov 2023 21:20:51 +0000 Received: from mail-ed1-x533.google.com ([2a00:1450:4864:20::533]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r8RzE-009JDW-0U for linux-arm-kernel@lists.infradead.org; Wed, 29 Nov 2023 21:20:49 +0000 Received: by mail-ed1-x533.google.com with SMTP id 4fb4d7f45d1cf-548f0b7ab11so55309a12.1 for ; Wed, 29 Nov 2023 13:20:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701292844; x=1701897644; darn=lists.infradead.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=e7uJ6/cKIhORoXu8TzZSGHhq752ZWeeIg37a8KHIARo=; b=TNz01NFPhB/9uVxaok7rvif6qoUvCaoq+Z0smQa7nWBxK+44UZcm7nBo0avrzsTphv zpsSbmegxuq/JO79L4IF1rhJ66iK23C5mr3kKIHPATT/lRrnrZmwvF5k3YKy7yHSmm4a id3QJyxMF1hoyKmAQVBuE9A4svKIE0eViQ7wuk9X+mdPIZu17FM8/zEMeZHvQm4Acqk8 /GjshHTY8o3US+vZ+83NtEbtp+qxu6yLo+0GanodoXIX5apjdHfZTuJlqZpg1tyukFOz eBDtlrTgbGzIcKq0hxTdRgcOKHSjEy0vh0xt6v6qE8U+rDqzDO57vVoXRrPQ5k7yOa6b ivFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701292844; x=1701897644; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=e7uJ6/cKIhORoXu8TzZSGHhq752ZWeeIg37a8KHIARo=; b=j2hrWj72TQ6OweO+msxC7/dUev6og2s1HURaf23cngvTHWzQVgdOyYJ3FU4AraXO+3 BriynCUqRX/LXU0U/DqTJvboOqp/3Pbo84HnBbTgAOn/s+cxYtsDzt/kCSTj+ka7qNof e1+BwE9qPvOXAn4ESOBFzoq0NZZO5FGv3vCuAS0kicdxYQxaP+yZjTHUVE38UTZMC8oE ozza5iXJcIMD/0oRV4rxWCXoZtfGMt9WJs0XBWOypBanB13FcMXqTZOc4zD5hlC9/3Ji wrdeSmt2TVajxKTSD8vrnaD+Bv94NsVLp6H6mlaBAoQZ1McmLC8AVbcxPR44onXNphHk J1tA== X-Gm-Message-State: AOJu0Yxt3xjiCT6me6xE0ScQ3EJZevP47uhnLX9Oqz8W1vQxVx5s0WMF W2vkisgREqGOnTwCRqVeIe0= X-Google-Smtp-Source: AGHT+IHW256ZYrusTlNvNtiLRMCdgMj6p64wZO364+KYmDxL9yXjM8VrqXHYyIuRwwYq8Ld2nVxSCA== X-Received: by 2002:a17:906:27d9:b0:a01:c04a:ca9b with SMTP id k25-20020a17090627d900b00a01c04aca9bmr14609716ejc.72.1701292843807; Wed, 29 Nov 2023 13:20:43 -0800 (PST) Received: from [192.168.26.149] (031011218106.poznan.vectranet.pl. [31.11.218.106]) by smtp.googlemail.com with ESMTPSA id p18-20020a1709061b5200b009b9a1714524sm8409276ejg.12.2023.11.29.13.20.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Nov 2023 13:20:41 -0800 (PST) Message-ID: Date: Wed, 29 Nov 2023 22:20:38 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: ARM BCM53573 SoC hangs/lockups caused by locks/clock/random changes To: Linus Walleij Cc: Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Russell King , Daniel Lezcano , Thomas Gleixner , Florian Fainelli , linux-clk@vger.kernel.org, linux-arm-kernel@lists.infradead.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, openwrt-devel@lists.openwrt.org, bcm-kernel-feedback-list@broadcom.com References: Content-Language: en-US From: =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231129_132048_190760_484E7031 X-CRM114-Status: GOOD ( 22.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org SGksCgppdCdzIGEgbGF0ZSByZXBseSBidXQgSSBkaWRuJ3QgZmluZCBlbm91Z2ggZGV0ZXJtaW5h dGlvbiBlYXJsaWVyLgoKT24gOC4wOS4yMDIzIDEwOjEwLCBMaW51cyBXYWxsZWlqIHdyb3RlOgo+ IE9uIE1vbiwgU2VwIDQsIDIwMjMgYXQgMTA6MzTigK9BTSBSYWZhxYIgTWnFgmVja2kgPHphamVj NUBnbWFpbC5jb20+IHdyb3RlOgo+IAo+PiBJJ20gY2x1ZWxlc3MgYXQgdGhpcyBwb2ludC4KPj4g TWF5YmUgc29tZW9uZSBjYW4gY29tZSB1cCB3aXRoIGFuIGlkZWEgb2YgYWN0dWFsIGlzc3VlICYg aWRlYWxseSBhCj4+IHNvbHV0aW9uLgo+IAo+IERhbW4gdGhpcyBpcyBmcnVzdHJhdGluZy4KPiAK Pj4gMi4gQ2xvY2sgKGFybSxhcm12Ny10aW1lcikKPj4KPj4gV2hpbGUgY29tcGFyaW5nIG1haW4g Y2xvY2sgaW4gQnJvYWRjb20ncyBTREsgd2l0aCB1cHN0cmVhbSBvbmUgSSBub3RpY2VkCj4+IGEg dGlueSBkaWZmZXJlbmNlOiBtYXNrIHZhbHVlLiBJIGRvbid0IGtub3cgaXQgaXQgbWFrZXMgYW55 IHNlbnNlIGJ1dAo+PiBzd2l0Y2hpbmcgZnJvbSBDTE9DS1NPVVJDRV9NQVNLKDU2KSB0byBDTE9D S1NPVVJDRV9NQVNLKDY0KSBpbgo+PiBhcm1fYXJjaF90aW1lci5jICh0byBtYXRjaCBTREspIGlu Y3JlYXNlcyBhdmVyYWdlIHVwdGltZSAodGltZSBiZWZvcmUgYQo+PiBoYW5nL2xvY2t1cCBoYXBw ZW5zKSBmcm9tIDQgbWludXRlcyB0byAzNiBtaW51dGVzLgo+IAo+IFRoaXMgY291bGQgYmUgcmVs YXRlZCB0byBob3cgb2Z0ZW4gdGhlIHN5c3RlbSBnb2VzIHRvIGlkbGUuCj4gCj4+ICsgICAgICAg aWYgKGNwdV9pZGxlX2ZvcmNlX3BvbGwgPT0gMTIzNCkKPj4gKyAgICAgICAgICAgICAgIGFyY2hf Y3B1X2lkbGUoKTsKPj4gKyAgICAgICBpZiAoY3B1X2lkbGVfZm9yY2VfcG9sbCA9PSA1Njc4KQo+ PiArICAgICAgICAgICAgICAgYXJjaF9jcHVfaWRsZSgpOwo+PiArICAgICAgIGlmIChjcHVfaWRs ZV9mb3JjZV9wb2xsID09IDEyMzQpCj4+ICsgICAgICAgICAgICAgICBhcmNoX2NwdV9pZGxlKCk7 Cj4+ICsgICAgICAgaWYgKGNwdV9pZGxlX2ZvcmNlX3BvbGwgPT0gNTY3OCkKPj4gKyAgICAgICAg ICAgICAgIGFyY2hfY3B1X2lkbGUoKTsKPj4gKyAgICAgICBpZiAoY3B1X2lkbGVfZm9yY2VfcG9s bCA9PSAxMjM0KQo+PiArICAgICAgICAgICAgICAgYXJjaF9jcHVfaWRsZSgpOwo+PiArICAgICAg IGlmIChjcHVfaWRsZV9mb3JjZV9wb2xsID09IDU2NzgpCj4+ICsgICAgICAgICAgICAgICBhcmNo X2NwdV9pZGxlKCk7Cj4+ICsgICAgICAgaWYgKGNwdV9pZGxlX2ZvcmNlX3BvbGwgPT0gMTIzNCkK Pj4gKyAgICAgICAgICAgICAgIGFyY2hfY3B1X2lkbGUoKTsKPiAKPiBJZGxlIGFnYWluLgo+IAo+ IEkgd291bGQgaGF2ZSB0cmllZCB0byBzZWUgd2hhdCBhcmNoX2NwdV9pZGxlKCkgaXMgZG9pbmcu Cj4gCj4gYXJtX3BtX2lkbGUoKSBvciBjcHVfZG9faWRsZSgpPwoKSW4gbXkgY2FzZSBhcm1fcG1f aWRsZSBpcyBOVUxMLgoKCj4gV2hhdCBoYXBwZW5zIGlmIHlvdSBqdXN0IHB1dCByZXR1cm4gaW4g YXJjaF9jcHVfaWRsZSgpCj4gc28gaXQgZG9lcyBub3RoaW5nPwoKRG9lc24ndCBoZWxwLiBJIGFs c28gdHJpZWQgcHV0dGluZzoKdWRlbGF5KDEwKTsKYW5kCnVkZWxheSgxMDAwKTsKYXQgdGhlIGFy Y2hfY3B1X2lkbGUoKSBiZWdpbm5pbmcuIE5vbmUgaGVscGVkLgoKCkhlcmUgY29tZXMgbW9yZSBp bnRlcmVzdGluZyBleHBlcmltZW50IHRob3VnaC4gUHV0dGluZyB0aGVyZToKCmlmICghKGZvbysr ICUgMTAwMDApKSB7Cglwcl9pbmZvKCJbJXNdIGFybV9wbV9pZGxlOiVwc1xuIiwgX19mdW5jX18s IGFybV9wbV9pZGxlKTsKfQoKZG9lc24ndCBzZWVtIHRvIGhlbHAuCgoKUHV0dGluZyBmb2xsb3dp bmcgaG93ZXZlciBzZWVtcyB0byBtYWtlIGtlcm5lbC9kZXZpY2Ugc3RhYmxlOgoKaWYgKCEoZm9v KysgJSAxMDApKSB7Cglwcl9pbmZvKCJbJXNdIGFybV9wbV9pZGxlOiVwc1xuIiwgX19mdW5jX18s IGFybV9wbV9pZGxlKTsKfQoKCkkgdGhpbmsgSSdtIGp1c3QgZ29pbmcgdG8gYXNzdW1lIHRob3Nl IGNoaXBzZXRzIGFyZSBzaW1wbHkgaHcgYnJva2VuLgoKX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QKbGlu dXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQu b3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=