From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3024917E015 for ; Mon, 20 Jan 2025 07:37:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737358668; cv=none; b=PsXmxB9gCuV3kA9cZyJZlxN0hl3Zo4POzfFOa5rRFb7HD5WZs6Tan15MGTotie0tUU+a513+j8oAJ9FU1kspWJ0u6eZP/rHlw0GI3WK4+wZkeRmaQLeB1Kdzn/+GqJltk8TKekbBUYW0o8B/jQeKlBheNQQxdvFGIQAVwwyjEww= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737358668; c=relaxed/simple; bh=g97ouC4g7MwYoFfldxsEV0MPS1/iaaL8LhaR7jYH7GU=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=XB+3vHqrSOrTiMpdC5xaHSvGhArTAS0IUUA/R5ZhM0qwBNnAbbLTvns0FKFWIzh5e006j3C3yK014EnXZEvWE9Y9NUxJ20lhU+7iau0ksmjJNQSaHQPByFXDcLtbuw4o55HszH5TpEeGD0Ls2sfGAM1oNHxf0ui0kDX9BlLntwA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=UuMQ4OPU; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=T6Ah/KrL; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="UuMQ4OPU"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="T6Ah/KrL" From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1737358658; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=94gHCrclhJ8NEhggf/2SeD20YartfOWGhKHSTZJsRIE=; b=UuMQ4OPUchjamOeDlCKd556Bxya70t/9132nejDOfsXoimV7Fw66XRvBAwsqYuuFdhBbi2 MbZSXGDDKqM0hVAp0lM9T8PsMjhpg22XMl+VpLPFPlpS6pod95LXRjj1feK5REozGpoMlN Sgfh9DlnmxyrD2t/9Dhgtm6YSemVkri/rDnZgpYLt73PPQlkJRih7LvlpuL12OrJJpuJVM /ZN5DX1SXF2D0VYxR1CDbPBOqfZ35ON6vrLAyit0vGECvk8VqNu632KgumFarR4QXmnY4U Jr3MZKyXR4QLiCMDDLITeuCVi0gsnsCY2QY4aGKP4CBI3TfgTV0hw1nP7xmjHA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1737358658; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=94gHCrclhJ8NEhggf/2SeD20YartfOWGhKHSTZJsRIE=; b=T6Ah/KrLDv/nwjv0i4LgCXfFvrDagPr6fUdKKnRZfidUOcKQnOPmx5inC+4F3EGkkENE6p Qmsn4YUCHJ7QCeCg== To: Anup Patel Cc: Charlie Jenkins , Xu Lu , paul.walmsley@sifive.com, palmer@dabbelt.com, lihangjing@bytedance.com, xieyongji@bytedance.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH] irqchip: riscv: Order normal writes and IPI writes In-Reply-To: References: <20250116120710.51673-1-luxu.kernel@bytedance.com> <87ldv9afso.ffs@tglx> Date: Mon, 20 Jan 2025 08:37:37 +0100 Message-ID: <87frle9br2.ffs@tglx> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Fri, Jan 17 2025 at 21:23, Anup Patel wrote: > On Fri, Jan 17, 2025 at 4:05=E2=80=AFPM Thomas Gleixner wrote: >> On Thu, Jan 16 2025 at 13:09, Charlie Jenkins wrote: >> > On Thu, Jan 16, 2025 at 08:07:10PM +0800, Xu Lu wrote: >> >> Replace writel_relaxed() with writel() when issuing IPI to ensure all >> >> previous write operations made by current CPU are visible to other CP= Us. >> > >> > Did you experience an ordering issue from this? >> >> That's not the right question. >> >> CPU 0 CPU 1 >> store A // data >> store B // IPI >> IPI handler >> load A >> >> The real question is whether the RISC-V memory model guarantees under >> all circumstances that A is globally visible before the IPI handler >> load. If so, then the writel_relaxed() is fine. If not, the fence is >> required. >> >> That's not a question of observation. It's a question of facts defined >> by the architecture. People have wasted months to analyze such fails >> which tend to happen once every blue moon with no other trace than >> "something went wrong" .... > > The RISC-V FENCE instruction distinguishes between normal > memory operations and I/O operations in its predecessor and > successor sets where r =3D normal read, w =3D normal write, > i =3D I/O read, and o =3D I/O write. > > The ipi_mux_send_mask() does smp_mb__after_atomic() (equals > to "fence rw,rw") before calling imsic_ipi_send(). This prevents > ordering of normal memory writes in imsic_ipi_send() before > smp_mb__after_atomic() in ipi_mux_send_mask() but it does > not prevent I/O memory writes in imsic_ipi_send() to be ordered > before smp_mb__after_atomic(). > > This means currently nothing prevents the I/O memory write in > imsic_ipi_send() to be ordered before normal memory writes in > ipi_mux_send_mask() hence there is a very unlikely possibility > of an IPI handler on the target CPU seeing incorrect data. Very unlikely is a valid assumption for a single system, but at scale it becomes very likely :) > The conversion of writel_relaxed() to writel() in imsic_ipi_send() > adds a "fence w,o" before the actual I/O memory write which > ensures that I/O memory write is not ordered before normal > memory writes. > > Based on the above, the conversion of writel_relaxed() to > writel() in imsic_ipi_send() looks good to me. Can we please have something like the above in the change log so this is documented for posterity? Thanks tglx