From mboxrd@z Thu Jan 1 00:00:00 1970 From: Corey Minyard Date: Wed, 21 Feb 2024 12:08:16 -0600 Subject: [PATCH] ipmi: kcs: Update OBF poll timeout to reduce latency In-Reply-To: <527F52AB-0070-43EA-BE82-945280CA2AEE@gmail.com> References: <20240220123615.963916-1-geissonator@gmail.com> <9680ad7d7a48fc36a0572dc2286a1229a29341fe.camel@codeconstruct.com.au> <527F52AB-0070-43EA-BE82-945280CA2AEE@gmail.com> Message-ID: List-Id: To: linux-aspeed@lists.ozlabs.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Wed, Feb 21, 2024 at 10:57:38AM -0600, Andrew Geissler wrote: > > > > On Feb 20, 2024, at 4:36?PM, Andrew Jeffery wrote: > > > > On Tue, 2024-02-20 at 13:33 -0600, Corey Minyard wrote: > >> On Tue, Feb 20, 2024 at 04:51:21PM +0100, Paul Menzel wrote: > >>> Dear Andrew, > >> > >> It's because increasing that number causes it to poll longer for the > >> event, the host takes longer than 100us to generate the event, and if > >> the event is missed the time when it is checked again is very long. > >> > >> Polling for 100us is already pretty extreme. 200us is really too long. > >> > >> The real problem is that there is no interrupt for this. I'd also guess > >> there is no interrupt on the host side, because that would solve this > >> problem, too, as it would certainly get around to handling the interupt > >> in 100us. I'm assuming the host driver is not the Linux driver, as it > >> should also handle this in a timely manner, even when polling. > > > > I expect the issues Andrew G is observing are with the Power10 boot > > firmware. The boot firmware only polls. The runtime firmware enables > > interrupts. > > Yep, this is with the low level host boot firmware. > Also, further testing over night showed that 200us wasn?t enough for > our larger Everest P10 machines, I needed to go to 300us. As we > were struggling to allow 200us, I assume 300us is going to be a no-go. It seems odd to me that firmware polling would be an issue. Usually, with firmware, you have it just spinning waiting for something. At least in the firmware I worked with. I'm not familiar with this firmware, though, maybe it has timers and such and parallel execution. Can this be fixed on the firmware side? > > >> > > > >> > >> The right way to fix this is probably to do the same thing the host side > >> Linux driver does. It has a kernel thread that is kicked off to do > >> this. Unfortunately, that's more complicated to implement, but it > >> avoids polling in this location (which causes latency issues on the BMC > >> side) and lets you poll longer without causing issues. > > > > In Andrew G's case he's talking MCTP over KCS using a vendor-defined > > transport binding (that also leverages LPC FWH cycles for bulk data > > transfers)[1]. I think it could have taken more inspiration from the > > IPMI KCS protocol: It might be worth an experiment to write the dummy > > command value to IDR from the host side after each ODR read to signal > > the host's clearing of OBF (no interrupt for the BMC) with an IBF > > (which does interrupt the BMC). And doing the obverse for the BMC. Some > > brief thought suggests that if the dummy value is read there's no need > > to send a dummy value in reply (as it's an indicator to read the status > > register). With that the need for the spin here (or on the host side) > > is reduced at the cost of some constant protocol overhead. > > > > Thanks for the quick reviews and ideas. > I?ll see if I can find someone on the team to help out with Andrew J?s > thoughts and if that doesn?t work, look into the kernel thread idea. I don't really understand Andrew J's ideas very well, but hopefully they help. The kernel thread idea is fairly complicated to implement, and there has been an impetus in the kernel to not create new kernel threads. But there just has to be a good reason, and this probably is one. We worked on it a lot in the IPMI host driver to tune it and got it to a point where it provided decent performance without causing power management issues. When I first read the title I was worried it was talking about this code; I'm lothe to touch it for fear of breaking things. -corey From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 935D9C5478A for ; Wed, 21 Feb 2024 18:09:25 +0000 (UTC) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=bS7ByvFi; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Tg46X0Mj0z3d2d for ; Thu, 22 Feb 2024 05:09:24 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=bS7ByvFi; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::c30; helo=mail-oo1-xc30.google.com; envelope-from=tcminyard@gmail.com; receiver=lists.ozlabs.org) Received: from mail-oo1-xc30.google.com (mail-oo1-xc30.google.com [IPv6:2607:f8b0:4864:20::c30]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Tg45R2mlHz3cRd; Thu, 22 Feb 2024 05:08:25 +1100 (AEDT) Received: by mail-oo1-xc30.google.com with SMTP id 006d021491bc7-59fe5b77c0cso495670eaf.0; Wed, 21 Feb 2024 10:08:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708538900; x=1709143700; darn=lists.ozlabs.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:reply-to:message-id:subject:cc:to:from:date :sender:from:to:cc:subject:date:message-id:reply-to; bh=yE3xYE9b0PADKF8ltORqAx3XVvlJ9P1Y3jeB29+/qOg=; b=bS7ByvFi2QZvPnqOThsyxvog50jv2Biwpvfd5DvCqJSdDBzHiG3/pd51ROCPaz4si0 NWXQ3xESeENicoGN3NaittVe2I8Fm3jJMTFMuNj7PvGgq2UQNB/ZH+z3YENNgRBtWZig g+aXVjQxo0MeHQ4tdMTOEozmzB0bpBnn0Pg3awJLF3lGxxtwC/8e4SeSyCLyFPC4eVBh RkF5/7HJvDG9nUhGp0K3E4k499hA5E7yCsFLZN/U8B95ecR/0WkbvSRB/WEecGo04nNY yTy2Uo39uZXpO+aMzdKELG1YQVGMB3lfni8d5GZIA8PcFRPvoBJjVOVxrEkZYUcy/xHA XbUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708538900; x=1709143700; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:reply-to:message-id:subject:cc:to:from:date :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=yE3xYE9b0PADKF8ltORqAx3XVvlJ9P1Y3jeB29+/qOg=; b=MZ1towLG9gd3lGf32JYZ4Vgn7We/YZfykkUKyRuzAh10ucIaeRJfh/30pQRpu8QdRI dwTFUsm3JRwncehGZrE7m4aQrkRhB0W9ZKYXdZClAAxkGLIrYD0w+LbSiYWwp73mft6G pqqMv5tGGojzW7lWIZR9yFqsCqGJGHVTC9Rfl/Szr3qtiKnR3G3qbSQMaNbxlQQLVPPC rWF9GR6iCeixys/2eyF6YJjoezhSu5mBZj4Gjpi0IsMz+un+nzANX/jr+3smTK9Mk4g6 X8rHcz1hX/qfqIVsJ0XkIpzJ4vnb0F0X6xbOwsXnncWZZBFkm7hGtdKFpyW8nLxzav8Z MBJw== X-Forwarded-Encrypted: i=1; AJvYcCX9Wqxqkj93H6WoeG/2EXbZMy7W4d0VdEXU1LKoTtGJ3MaWfZNyHbVSstMohv+1spKV48JUp1raT+FafC1Ma3ZCG7s6lFYNQEwL+Bx5cbEjkLImw5fhedzXKO1KZohcqLygJRCvMcUn X-Gm-Message-State: AOJu0YxANfgKT42H2893g9GdmxyUTN8nhHHwekgWXy1I6zd6MKR4nhsN 5d3CHG13Yu9HmVRlFus++NGxm3aN3fiMCbwqQiYsMg5yXbnbc84= X-Google-Smtp-Source: AGHT+IE7Jl89TMdslT9Xstr1epQWCcXolp3sqaBdeNYzQrNiseiNxKWZkq7GYlWCsP8bfbZaGYK6yg== X-Received: by 2002:a4a:d2ce:0:b0:5a0:2a9:574b with SMTP id j14-20020a4ad2ce000000b005a002a9574bmr5993289oos.9.1708538899972; Wed, 21 Feb 2024 10:08:19 -0800 (PST) Received: from serve.minyard.net (serve.minyard.net. [2001:470:b8f6:1b::1]) by smtp.gmail.com with ESMTPSA id g6-20020a9d6206000000b006e2df00aaa8sm1665305otj.70.2024.02.21.10.08.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 10:08:19 -0800 (PST) Received: from mail.minyard.net (unknown [IPv6:2001:470:b8f6:1d::35]) by serve.minyard.net (Postfix) with ESMTPSA id 412431800B8; Wed, 21 Feb 2024 18:08:18 +0000 (UTC) Date: Wed, 21 Feb 2024 12:08:16 -0600 From: Corey Minyard To: Andrew Geissler Subject: Re: [PATCH] ipmi: kcs: Update OBF poll timeout to reduce latency Message-ID: References: <20240220123615.963916-1-geissonator@gmail.com> <9680ad7d7a48fc36a0572dc2286a1229a29341fe.camel@codeconstruct.com.au> <527F52AB-0070-43EA-BE82-945280CA2AEE@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <527F52AB-0070-43EA-BE82-945280CA2AEE@gmail.com> X-BeenThere: openbmc@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development list for OpenBMC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: minyard@acm.org Cc: Paul Menzel , linux-aspeed , openbmc@lists.ozlabs.org, Linux Kernel Mailing List , Joel Stanley , openipmi-developer@lists.sourceforge.net, Linux ARM Errors-To: openbmc-bounces+openbmc=archiver.kernel.org@lists.ozlabs.org Sender: "openbmc" On Wed, Feb 21, 2024 at 10:57:38AM -0600, Andrew Geissler wrote: > > > > On Feb 20, 2024, at 4:36 PM, Andrew Jeffery wrote: > > > > On Tue, 2024-02-20 at 13:33 -0600, Corey Minyard wrote: > >> On Tue, Feb 20, 2024 at 04:51:21PM +0100, Paul Menzel wrote: > >>> Dear Andrew, > >> > >> It's because increasing that number causes it to poll longer for the > >> event, the host takes longer than 100us to generate the event, and if > >> the event is missed the time when it is checked again is very long. > >> > >> Polling for 100us is already pretty extreme. 200us is really too long. > >> > >> The real problem is that there is no interrupt for this. I'd also guess > >> there is no interrupt on the host side, because that would solve this > >> problem, too, as it would certainly get around to handling the interupt > >> in 100us. I'm assuming the host driver is not the Linux driver, as it > >> should also handle this in a timely manner, even when polling. > > > > I expect the issues Andrew G is observing are with the Power10 boot > > firmware. The boot firmware only polls. The runtime firmware enables > > interrupts. > > Yep, this is with the low level host boot firmware. > Also, further testing over night showed that 200us wasn’t enough for > our larger Everest P10 machines, I needed to go to 300us. As we > were struggling to allow 200us, I assume 300us is going to be a no-go. It seems odd to me that firmware polling would be an issue. Usually, with firmware, you have it just spinning waiting for something. At least in the firmware I worked with. I'm not familiar with this firmware, though, maybe it has timers and such and parallel execution. Can this be fixed on the firmware side? > > >> > > > >> > >> The right way to fix this is probably to do the same thing the host side > >> Linux driver does. It has a kernel thread that is kicked off to do > >> this. Unfortunately, that's more complicated to implement, but it > >> avoids polling in this location (which causes latency issues on the BMC > >> side) and lets you poll longer without causing issues. > > > > In Andrew G's case he's talking MCTP over KCS using a vendor-defined > > transport binding (that also leverages LPC FWH cycles for bulk data > > transfers)[1]. I think it could have taken more inspiration from the > > IPMI KCS protocol: It might be worth an experiment to write the dummy > > command value to IDR from the host side after each ODR read to signal > > the host's clearing of OBF (no interrupt for the BMC) with an IBF > > (which does interrupt the BMC). And doing the obverse for the BMC. Some > > brief thought suggests that if the dummy value is read there's no need > > to send a dummy value in reply (as it's an indicator to read the status > > register). With that the need for the spin here (or on the host side) > > is reduced at the cost of some constant protocol overhead. > > > > Thanks for the quick reviews and ideas. > I’ll see if I can find someone on the team to help out with Andrew J’s > thoughts and if that doesn’t work, look into the kernel thread idea. I don't really understand Andrew J's ideas very well, but hopefully they help. The kernel thread idea is fairly complicated to implement, and there has been an impetus in the kernel to not create new kernel threads. But there just has to be a good reason, and this probably is one. We worked on it a lot in the IPMI host driver to tune it and got it to a point where it provided decent performance without causing power management issues. When I first read the title I was worried it was talking about this code; I'm lothe to touch it for fear of breaking things. -corey From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 68C14C5478A for ; Wed, 21 Feb 2024 18:08:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Reply-To:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=eh5WPWVcHQJ/7POYkJ5h+OjiFDvqu3vsjDLcXWririo=; b=BLjisMOyOrjMfv JU+hfKVYrODovoAlmiGDyksQf/kp9R+CbdevnXn1kGPyIz3FiCBUtqrPreDwC1b9elcgXKxgBsXsb pdyxkhBclEDf0RB0d0ygakXPnN1uiyyl7gip9/jZ4wWdvPMYmVAyZ63dDk9NZLU39bbsX8DVVZVNs hfL8De1khSvkY8Te0wCNnIL1giAE/Z6QZYLTXqaVtgeGeFT1PduHqVY8kdjNSJyOP1Siqfqu125GP luM9oMrdBXCotRqN6FgIvMZ+zYr7Xt3vXqSzRQCG9UUSAGWv7MQ9OQ7JnTxAVEb7NlRhylZeM1lfe CzNd5cflLftzni6j5/Lg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rcr17-000000021MQ-2ctz; Wed, 21 Feb 2024 18:08:25 +0000 Received: from mail-oo1-xc2e.google.com ([2607:f8b0:4864:20::c2e]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rcr15-000000021M0-09Dk for linux-arm-kernel@lists.infradead.org; Wed, 21 Feb 2024 18:08:24 +0000 Received: by mail-oo1-xc2e.google.com with SMTP id 006d021491bc7-59f7d59d3f1so454235eaf.1 for ; Wed, 21 Feb 2024 10:08:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708538900; x=1709143700; darn=lists.infradead.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:reply-to:message-id:subject:cc:to:from:date :sender:from:to:cc:subject:date:message-id:reply-to; bh=yE3xYE9b0PADKF8ltORqAx3XVvlJ9P1Y3jeB29+/qOg=; b=UjucYTvQZnN/AFWSXk/onRGmWzjzCbzW6PpmcINKu3fF70yRh2mRQkJGX4lQ/4zSbP E/zBa/TCumuS346Laahptlca7wZBbIf/V+GSYW70gI+O6pV0hvupE0im6AmffShxXwv5 ZrQtdgslUiCjNE4tjoF2njalP1RW+kwmqWlqv36j53i5vclPEkBEMgHtrfHardMVUGMB RDBGCBc33VqJNOauedQCqMC2r+Pr+1tRiyMVnW4ZmwPlfcko9/Ou3nWC0HvgltN78BmP T8JATAczBmMIXMNS9KatvHF15NLHEAToDcsp3x1s1yaB9+HwqVuJL4V3rzMD1fjAkj4d 6MJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708538900; x=1709143700; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:reply-to:message-id:subject:cc:to:from:date :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=yE3xYE9b0PADKF8ltORqAx3XVvlJ9P1Y3jeB29+/qOg=; b=JM1Y8WikngLMF7R3Z4R0zLOwRVkrpKncZfzVTSZnRqqKF+M6I8/TyNcfCmuHabbsin IdENUJOxLhRgyBCaQBeFsIG8WszoZcgjkDJVgdKDiBUv5IFqKXlj0So2Vvu3qcIvfo0R vMF3nObo2RyjinGM4TVFUph8ouSf5CESQYC0KAqoxSGgFSF4x5ChSkhUHN5KTGhxulBy kINrq1efWXN+3GqrpN9Lv+MwLvB3sOW3yF2IMBxr/5Wm5zxTlFJpJJqw6Z3mfY5njPUn comYa1bUd9pPTQZqtlFUAyRU3KIMTLJB6H5HoD22L0ANazzAfbnXYEa/hfBKAo0mb1Mv +zeg== X-Forwarded-Encrypted: i=1; AJvYcCV/cRC4uNv1srQTLkY1pKpBuL8L9PJZhiboSdkIuLs1cV49NrtThBQkFcEJwoKRTn3rQF3mSfu0a7PMPXvRU8u75P1UZHtz5eko8ILHL8hQgBcTyGw= X-Gm-Message-State: AOJu0YxALkCYimQX1zraXIaPtz5wUmDu9smcNJiTpPfAbNMxSAJ192k/ X8rDfbQPVxt426J0qSflz/EIECl7oJVVE80zvlhpH9lfdF86Blw= X-Google-Smtp-Source: AGHT+IE7Jl89TMdslT9Xstr1epQWCcXolp3sqaBdeNYzQrNiseiNxKWZkq7GYlWCsP8bfbZaGYK6yg== X-Received: by 2002:a4a:d2ce:0:b0:5a0:2a9:574b with SMTP id j14-20020a4ad2ce000000b005a002a9574bmr5993289oos.9.1708538899972; Wed, 21 Feb 2024 10:08:19 -0800 (PST) Received: from serve.minyard.net (serve.minyard.net. [2001:470:b8f6:1b::1]) by smtp.gmail.com with ESMTPSA id g6-20020a9d6206000000b006e2df00aaa8sm1665305otj.70.2024.02.21.10.08.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 10:08:19 -0800 (PST) Received: from mail.minyard.net (unknown [IPv6:2001:470:b8f6:1d::35]) by serve.minyard.net (Postfix) with ESMTPSA id 412431800B8; Wed, 21 Feb 2024 18:08:18 +0000 (UTC) Date: Wed, 21 Feb 2024 12:08:16 -0600 From: Corey Minyard To: Andrew Geissler Cc: Andrew Jeffery , Paul Menzel , Joel Stanley , openipmi-developer@lists.sourceforge.net, Linux ARM , linux-aspeed , Linux Kernel Mailing List , openbmc@lists.ozlabs.org Subject: Re: [PATCH] ipmi: kcs: Update OBF poll timeout to reduce latency Message-ID: References: <20240220123615.963916-1-geissonator@gmail.com> <9680ad7d7a48fc36a0572dc2286a1229a29341fe.camel@codeconstruct.com.au> <527F52AB-0070-43EA-BE82-945280CA2AEE@gmail.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <527F52AB-0070-43EA-BE82-945280CA2AEE@gmail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240221_100823_098745_02F7B37B X-CRM114-Status: GOOD ( 43.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: minyard@acm.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gV2VkLCBGZWIgMjEsIDIwMjQgYXQgMTA6NTc6MzhBTSAtMDYwMCwgQW5kcmV3IEdlaXNzbGVy IHdyb3RlOgo+IAo+IAo+ID4gT24gRmViIDIwLCAyMDI0LCBhdCA0OjM24oCvUE0sIEFuZHJldyBK ZWZmZXJ5IDxhbmRyZXdAY29kZWNvbnN0cnVjdC5jb20uYXU+IHdyb3RlOgo+ID4gCj4gPiBPbiBU dWUsIDIwMjQtMDItMjAgYXQgMTM6MzMgLTA2MDAsIENvcmV5IE1pbnlhcmQgd3JvdGU6Cj4gPj4g T24gVHVlLCBGZWIgMjAsIDIwMjQgYXQgMDQ6NTE6MjFQTSArMDEwMCwgUGF1bCBNZW56ZWwgd3Jv dGU6Cj4gPj4+IERlYXIgQW5kcmV3LAo+ID4+IAo+ID4+IEl0J3MgYmVjYXVzZSBpbmNyZWFzaW5n IHRoYXQgbnVtYmVyIGNhdXNlcyBpdCB0byBwb2xsIGxvbmdlciBmb3IgdGhlCj4gPj4gZXZlbnQs IHRoZSBob3N0IHRha2VzIGxvbmdlciB0aGFuIDEwMHVzIHRvIGdlbmVyYXRlIHRoZSBldmVudCwg YW5kIGlmCj4gPj4gdGhlIGV2ZW50IGlzIG1pc3NlZCB0aGUgdGltZSB3aGVuIGl0IGlzIGNoZWNr ZWQgYWdhaW4gaXMgdmVyeSBsb25nLgo+ID4+IAo+ID4+IFBvbGxpbmcgZm9yIDEwMHVzIGlzIGFs cmVhZHkgcHJldHR5IGV4dHJlbWUuIDIwMHVzIGlzIHJlYWxseSB0b28gbG9uZy4KPiA+PiAKPiA+ PiBUaGUgcmVhbCBwcm9ibGVtIGlzIHRoYXQgdGhlcmUgaXMgbm8gaW50ZXJydXB0IGZvciB0aGlz LiAgSSdkIGFsc28gZ3Vlc3MKPiA+PiB0aGVyZSBpcyBubyBpbnRlcnJ1cHQgb24gdGhlIGhvc3Qg c2lkZSwgYmVjYXVzZSB0aGF0IHdvdWxkIHNvbHZlIHRoaXMKPiA+PiBwcm9ibGVtLCB0b28sIGFz IGl0IHdvdWxkIGNlcnRhaW5seSBnZXQgYXJvdW5kIHRvIGhhbmRsaW5nIHRoZSBpbnRlcnVwdAo+ ID4+IGluIDEwMHVzLiAgSSdtIGFzc3VtaW5nIHRoZSBob3N0IGRyaXZlciBpcyBub3QgdGhlIExp bnV4IGRyaXZlciwgYXMgaXQKPiA+PiBzaG91bGQgYWxzbyBoYW5kbGUgdGhpcyBpbiBhIHRpbWVs eSBtYW5uZXIsIGV2ZW4gd2hlbiBwb2xsaW5nLgo+ID4gCj4gPiBJIGV4cGVjdCB0aGUgaXNzdWVz IEFuZHJldyBHIGlzIG9ic2VydmluZyBhcmUgd2l0aCB0aGUgUG93ZXIxMCBib290Cj4gPiBmaXJt d2FyZS4gVGhlIGJvb3QgZmlybXdhcmUgb25seSBwb2xscy4gVGhlIHJ1bnRpbWUgZmlybXdhcmUg ZW5hYmxlcwo+ID4gaW50ZXJydXB0cy4KPiAKPiBZZXAsIHRoaXMgaXMgd2l0aCB0aGUgbG93IGxl dmVsIGhvc3QgYm9vdCBmaXJtd2FyZS4KPiBBbHNvLCBmdXJ0aGVyIHRlc3Rpbmcgb3ZlciBuaWdo dCBzaG93ZWQgdGhhdCAyMDB1cyB3YXNu4oCZdCBlbm91Z2ggZm9yCj4gb3VyIGxhcmdlciBFdmVy ZXN0IFAxMCBtYWNoaW5lcywgSSBuZWVkZWQgdG8gZ28gdG8gMzAwdXMuIEFzIHdlCj4gd2VyZSBz dHJ1Z2dsaW5nIHRvIGFsbG93IDIwMHVzLCBJIGFzc3VtZSAzMDB1cyBpcyBnb2luZyB0byBiZSBh IG5vLWdvLgoKSXQgc2VlbXMgb2RkIHRvIG1lIHRoYXQgZmlybXdhcmUgcG9sbGluZyB3b3VsZCBi ZSBhbiBpc3N1ZS4gIFVzdWFsbHksCndpdGggZmlybXdhcmUsIHlvdSBoYXZlIGl0IGp1c3Qgc3Bp bm5pbmcgd2FpdGluZyBmb3Igc29tZXRoaW5nLiAgQXQKbGVhc3QgaW4gdGhlIGZpcm13YXJlIEkg d29ya2VkIHdpdGguCgpJJ20gbm90IGZhbWlsaWFyIHdpdGggdGhpcyBmaXJtd2FyZSwgdGhvdWdo LCBtYXliZSBpdCBoYXMgdGltZXJzIGFuZApzdWNoIGFuZCBwYXJhbGxlbCBleGVjdXRpb24uICBD YW4gdGhpcyBiZSBmaXhlZCBvbiB0aGUgZmlybXdhcmUgc2lkZT8KCj4gCj4gPj4gCj4gPiAKPiA+ PiAKPiA+PiBUaGUgcmlnaHQgd2F5IHRvIGZpeCB0aGlzIGlzIHByb2JhYmx5IHRvIGRvIHRoZSBz YW1lIHRoaW5nIHRoZSBob3N0IHNpZGUKPiA+PiBMaW51eCBkcml2ZXIgZG9lcy4gIEl0IGhhcyBh IGtlcm5lbCB0aHJlYWQgdGhhdCBpcyBraWNrZWQgb2ZmIHRvIGRvCj4gPj4gdGhpcy4gIFVuZm9y dHVuYXRlbHksIHRoYXQncyBtb3JlIGNvbXBsaWNhdGVkIHRvIGltcGxlbWVudCwgYnV0IGl0Cj4g Pj4gYXZvaWRzIHBvbGxpbmcgaW4gdGhpcyBsb2NhdGlvbiAod2hpY2ggY2F1c2VzIGxhdGVuY3kg aXNzdWVzIG9uIHRoZSBCTUMKPiA+PiBzaWRlKSBhbmQgbGV0cyB5b3UgcG9sbCBsb25nZXIgd2l0 aG91dCBjYXVzaW5nIGlzc3Vlcy4KPiA+IAo+ID4gSW4gQW5kcmV3IEcncyBjYXNlIGhlJ3MgdGFs a2luZyBNQ1RQIG92ZXIgS0NTIHVzaW5nIGEgdmVuZG9yLWRlZmluZWQKPiA+IHRyYW5zcG9ydCBi aW5kaW5nICh0aGF0IGFsc28gbGV2ZXJhZ2VzIExQQyBGV0ggY3ljbGVzIGZvciBidWxrIGRhdGEK PiA+IHRyYW5zZmVycylbMV0uIEkgdGhpbmsgaXQgY291bGQgaGF2ZSB0YWtlbiBtb3JlIGluc3Bp cmF0aW9uIGZyb20gdGhlCj4gPiBJUE1JIEtDUyBwcm90b2NvbDogSXQgbWlnaHQgYmUgd29ydGgg YW4gZXhwZXJpbWVudCB0byB3cml0ZSB0aGUgZHVtbXkKPiA+IGNvbW1hbmQgdmFsdWUgdG8gSURS IGZyb20gdGhlIGhvc3Qgc2lkZSBhZnRlciBlYWNoIE9EUiByZWFkIHRvIHNpZ25hbAo+ID4gdGhl IGhvc3QncyBjbGVhcmluZyBvZiBPQkYgKG5vIGludGVycnVwdCBmb3IgdGhlIEJNQykgd2l0aCBh biBJQkYKPiA+ICh3aGljaCBkb2VzIGludGVycnVwdCB0aGUgQk1DKS4gQW5kIGRvaW5nIHRoZSBv YnZlcnNlIGZvciB0aGUgQk1DLiBTb21lCj4gPiBicmllZiB0aG91Z2h0IHN1Z2dlc3RzIHRoYXQg aWYgdGhlIGR1bW15IHZhbHVlIGlzIHJlYWQgdGhlcmUncyBubyBuZWVkCj4gPiB0byBzZW5kIGEg ZHVtbXkgdmFsdWUgaW4gcmVwbHkgKGFzIGl0J3MgYW4gaW5kaWNhdG9yIHRvIHJlYWQgdGhlIHN0 YXR1cwo+ID4gcmVnaXN0ZXIpLiBXaXRoIHRoYXQgdGhlIG5lZWQgZm9yIHRoZSBzcGluIGhlcmUg KG9yIG9uIHRoZSBob3N0IHNpZGUpCj4gPiBpcyByZWR1Y2VkIGF0IHRoZSBjb3N0IG9mIHNvbWUg Y29uc3RhbnQgcHJvdG9jb2wgb3ZlcmhlYWQuCj4gPiAKPiAKPiBUaGFua3MgZm9yIHRoZSBxdWlj ayByZXZpZXdzIGFuZCBpZGVhcy4KPiBJ4oCZbGwgc2VlIGlmIEkgY2FuIGZpbmQgc29tZW9uZSBv biB0aGUgdGVhbSB0byBoZWxwIG91dCB3aXRoIEFuZHJldyBK4oCZcwo+IHRob3VnaHRzIGFuZCBp ZiB0aGF0IGRvZXNu4oCZdCB3b3JrLCBsb29rIGludG8gdGhlIGtlcm5lbCB0aHJlYWQgaWRlYS4K CkkgZG9uJ3QgcmVhbGx5IHVuZGVyc3RhbmQgQW5kcmV3IEoncyBpZGVhcyB2ZXJ5IHdlbGwsIGJ1 dCBob3BlZnVsbHkgdGhleQpoZWxwLiAgVGhlIGtlcm5lbCB0aHJlYWQgaWRlYSBpcyBmYWlybHkg Y29tcGxpY2F0ZWQgdG8gaW1wbGVtZW50LCBhbmQKdGhlcmUgaGFzIGJlZW4gYW4gaW1wZXR1cyBp biB0aGUga2VybmVsIHRvIG5vdCBjcmVhdGUgbmV3IGtlcm5lbAp0aHJlYWRzLiAgQnV0IHRoZXJl IGp1c3QgaGFzIHRvIGJlIGEgZ29vZCByZWFzb24sIGFuZCB0aGlzIHByb2JhYmx5IGlzCm9uZS4g IFdlIHdvcmtlZCBvbiBpdCBhIGxvdCBpbiB0aGUgSVBNSSBob3N0IGRyaXZlciB0byB0dW5lIGl0 IGFuZCBnb3QKaXQgdG8gYSBwb2ludCB3aGVyZSBpdCBwcm92aWRlZCBkZWNlbnQgcGVyZm9ybWFu Y2Ugd2l0aG91dCBjYXVzaW5nIHBvd2VyCm1hbmFnZW1lbnQgaXNzdWVzLiAgV2hlbiBJIGZpcnN0 IHJlYWQgdGhlIHRpdGxlIEkgd2FzIHdvcnJpZWQgaXQgd2FzCnRhbGtpbmcgYWJvdXQgdGhpcyBj b2RlOyBJJ20gbG90aGUgdG8gdG91Y2ggaXQgZm9yIGZlYXIgb2YgYnJlYWtpbmcKdGhpbmdzLgoK LWNvcmV5CgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwps aW51eC1hcm0ta2VybmVsIG1haWxpbmcgbGlzdApsaW51eC1hcm0ta2VybmVsQGxpc3RzLmluZnJh ZGVhZC5vcmcKaHR0cDovL2xpc3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51 eC1hcm0ta2VybmVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oo1-f43.google.com (mail-oo1-f43.google.com [209.85.161.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED82D33F7 for ; Wed, 21 Feb 2024 18:08:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.43 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708538902; cv=none; b=P5tSSvhS7t9cOxQPYADy8Wyu45a4QgN7e+wHuMi4h1ABH5mrmIlitIIDoU5nlwtQGyxEhNG7gFT8jQpNHxzyN1zDQt/FFQjSbEGz2+CRl8St1troJNcZHrq2G7A/0xU0jSNEZX6d8YSP0Zc+CBrQIbVtpSvSpnXPVrFsK2q/2b8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708538902; c=relaxed/simple; bh=1p/8NvuyZYpCVrfbqqK5Vg//TA2N3vASs0mwKJR9M4E=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=JYSWyotW0AnN40ZKpzb2FbMLRcSX4/06Ms+G9FXYqhGIrQU7KieXxIgELKwpL8bFgIghsqe4Wu01J6PWav/yz2ZPOsag/GlIsbmdIXTtwd36rwBELkqopE6c6S29Ox/tbKjWLQ251iIxG1slIetsUegrJzxxhobn8CFBAuzRaEU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=acm.org; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=KrYPdTNF; arc=none smtp.client-ip=209.85.161.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KrYPdTNF" Received: by mail-oo1-f43.google.com with SMTP id 006d021491bc7-59fe5b77c0cso495669eaf.0 for ; Wed, 21 Feb 2024 10:08:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708538900; x=1709143700; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:reply-to:message-id:subject:cc:to:from:date :sender:from:to:cc:subject:date:message-id:reply-to; bh=yE3xYE9b0PADKF8ltORqAx3XVvlJ9P1Y3jeB29+/qOg=; b=KrYPdTNFkodRXXKs8yXEMlhmZnBt1nm6XYeTal7c2A6HopzoVU5NQMcK/Uo1qpE5QJ cnQeDU1gT7cIOHJUrmF7VcyDeOAhBiyobIBLZu67lz/iRQpdKDEU5I/fYhIJUCvoJcVr CJd/vHJeFfv6IKCf8qfXE5XygrqjBLFZezjLkx1bC4Y9znqLuFDYka2D7B3CKrRaMb3W rYv11GZkWGfJBamkq2Rt5CR1imRgGFBIPvSrDBLkF7OfEUT/sIS01cQ1YFRdHS1ag5T/ 9Dd2tEM/zFpdF/VKZ5nfiabfRjoiTVlLNJoHRVseh869/AfNCISqy8W8s3fFOyB8w0y5 efug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708538900; x=1709143700; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:reply-to:message-id:subject:cc:to:from:date :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=yE3xYE9b0PADKF8ltORqAx3XVvlJ9P1Y3jeB29+/qOg=; b=drGHfZskDBvSz0W2P8VGat12uu011HAiFQK/on61fhHUPEaT/FVpt6yl4xoi11GJ10 bbnQMX5mLZya3PDR2vDLgb4gXSS5utRHbB0FVyYzIYSvnAqOviL04tQvWEjHGXmRIj3p UaiQ8w7zKrLGioTG6tKBmH9JRtNaJrzNfWTRQF9rI11K/UC9CKtiGpEc85ephyqKC9xC 4ZkJAYw8wqpRtKxcCfsd1syIKB/CVFP7xpclFNtCAkhhksN9loQJP+yQBEpUO7esJzy6 yZlcoo5rvF2C62dOm2E4607qcZ+COEF9GTdg8bm1iAfIYjnkpFLN2c6Hw37q5RkSWk7b mvXw== X-Forwarded-Encrypted: i=1; AJvYcCXIcnxd/jdRbBr+K3CCFbISD6DuqyrUMS4otB2m0O4TQRjSj5hCEjotQxkUuWZtGysNm13b5x+VpG0J9lXdZnvLMhqgyWqTRnjd3CwB X-Gm-Message-State: AOJu0YzI0o7DNYjOVtoz0GD3NXJJYC/TUZmkGQ3bbjZrkjsQ2s3elNsW 6Nrry2B3UWtjBuRPU0VgdxDjbnEuC+YbWtukcT6PYXIe4dhRP3A= X-Google-Smtp-Source: AGHT+IE7Jl89TMdslT9Xstr1epQWCcXolp3sqaBdeNYzQrNiseiNxKWZkq7GYlWCsP8bfbZaGYK6yg== X-Received: by 2002:a4a:d2ce:0:b0:5a0:2a9:574b with SMTP id j14-20020a4ad2ce000000b005a002a9574bmr5993289oos.9.1708538899972; Wed, 21 Feb 2024 10:08:19 -0800 (PST) Received: from serve.minyard.net (serve.minyard.net. [2001:470:b8f6:1b::1]) by smtp.gmail.com with ESMTPSA id g6-20020a9d6206000000b006e2df00aaa8sm1665305otj.70.2024.02.21.10.08.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 10:08:19 -0800 (PST) Sender: Corey Minyard Received: from mail.minyard.net (unknown [IPv6:2001:470:b8f6:1d::35]) by serve.minyard.net (Postfix) with ESMTPSA id 412431800B8; Wed, 21 Feb 2024 18:08:18 +0000 (UTC) Date: Wed, 21 Feb 2024 12:08:16 -0600 From: Corey Minyard To: Andrew Geissler Cc: Andrew Jeffery , Paul Menzel , Joel Stanley , openipmi-developer@lists.sourceforge.net, Linux ARM , linux-aspeed , Linux Kernel Mailing List , openbmc@lists.ozlabs.org Subject: Re: [PATCH] ipmi: kcs: Update OBF poll timeout to reduce latency Message-ID: Reply-To: minyard@acm.org References: <20240220123615.963916-1-geissonator@gmail.com> <9680ad7d7a48fc36a0572dc2286a1229a29341fe.camel@codeconstruct.com.au> <527F52AB-0070-43EA-BE82-945280CA2AEE@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <527F52AB-0070-43EA-BE82-945280CA2AEE@gmail.com> On Wed, Feb 21, 2024 at 10:57:38AM -0600, Andrew Geissler wrote: > > > > On Feb 20, 2024, at 4:36 PM, Andrew Jeffery wrote: > > > > On Tue, 2024-02-20 at 13:33 -0600, Corey Minyard wrote: > >> On Tue, Feb 20, 2024 at 04:51:21PM +0100, Paul Menzel wrote: > >>> Dear Andrew, > >> > >> It's because increasing that number causes it to poll longer for the > >> event, the host takes longer than 100us to generate the event, and if > >> the event is missed the time when it is checked again is very long. > >> > >> Polling for 100us is already pretty extreme. 200us is really too long. > >> > >> The real problem is that there is no interrupt for this. I'd also guess > >> there is no interrupt on the host side, because that would solve this > >> problem, too, as it would certainly get around to handling the interupt > >> in 100us. I'm assuming the host driver is not the Linux driver, as it > >> should also handle this in a timely manner, even when polling. > > > > I expect the issues Andrew G is observing are with the Power10 boot > > firmware. The boot firmware only polls. The runtime firmware enables > > interrupts. > > Yep, this is with the low level host boot firmware. > Also, further testing over night showed that 200us wasn’t enough for > our larger Everest P10 machines, I needed to go to 300us. As we > were struggling to allow 200us, I assume 300us is going to be a no-go. It seems odd to me that firmware polling would be an issue. Usually, with firmware, you have it just spinning waiting for something. At least in the firmware I worked with. I'm not familiar with this firmware, though, maybe it has timers and such and parallel execution. Can this be fixed on the firmware side? > > >> > > > >> > >> The right way to fix this is probably to do the same thing the host side > >> Linux driver does. It has a kernel thread that is kicked off to do > >> this. Unfortunately, that's more complicated to implement, but it > >> avoids polling in this location (which causes latency issues on the BMC > >> side) and lets you poll longer without causing issues. > > > > In Andrew G's case he's talking MCTP over KCS using a vendor-defined > > transport binding (that also leverages LPC FWH cycles for bulk data > > transfers)[1]. I think it could have taken more inspiration from the > > IPMI KCS protocol: It might be worth an experiment to write the dummy > > command value to IDR from the host side after each ODR read to signal > > the host's clearing of OBF (no interrupt for the BMC) with an IBF > > (which does interrupt the BMC). And doing the obverse for the BMC. Some > > brief thought suggests that if the dummy value is read there's no need > > to send a dummy value in reply (as it's an indicator to read the status > > register). With that the need for the spin here (or on the host side) > > is reduced at the cost of some constant protocol overhead. > > > > Thanks for the quick reviews and ideas. > I’ll see if I can find someone on the team to help out with Andrew J’s > thoughts and if that doesn’t work, look into the kernel thread idea. I don't really understand Andrew J's ideas very well, but hopefully they help. The kernel thread idea is fairly complicated to implement, and there has been an impetus in the kernel to not create new kernel threads. But there just has to be a good reason, and this probably is one. We worked on it a lot in the IPMI host driver to tune it and got it to a point where it provided decent performance without causing power management issues. When I first read the title I was worried it was talking about this code; I'm lothe to touch it for fear of breaking things. -corey