From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A7DF0CED266 for ; Tue, 8 Oct 2024 05:04:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:In-Reply-To:From:References:Cc:To:Subject:MIME-Version:Date: Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Pxi2ug8hQ1leGmc2gZxWeO2a5Yus7SNrn9aUqgKLhvY=; b=Ba1MpjQm9FShE25TTv8LxU/saA DOwwrLllTc/K3wxf5+sqocmcs2xI3nOw1K/r+HjnDZMvSUOetANHvfwsRX9cePXUXEvLui+OBwQ1j 9wAyhezLHcy1SJ9Rq493NCqrJX192vUJw3kF4xO/wFPIetrCGyJblQscjkzTFBjZ4dpARq+QnNJwI b+vIN3CC4Ae4XqCWqHFKvyP2nx1IQ7R4SVY7NEkC9aU879fpZUumyOf6TDbgi2ZNoq5+yGsk4zmEJ W/9hzEB6CJ2lHeyMnLDY+r0Ht/ReLB/iBnP9OwJ7QTl5FdP6GT+fQmaZcgsiDVvZH6K/9fhpHNWsh wqcnIMdA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1sy2O5-00000004WPg-0kpx; Tue, 08 Oct 2024 05:03:57 +0000 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1sy2Ml-00000004WMs-1uqW for linux-arm-kernel@lists.infradead.org; Tue, 08 Oct 2024 05:02:37 +0000 Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-20b7259be6fso57180835ad.0 for ; Mon, 07 Oct 2024 22:02:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1728363754; x=1728968554; darn=lists.infradead.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=Pxi2ug8hQ1leGmc2gZxWeO2a5Yus7SNrn9aUqgKLhvY=; b=bqiKYJNQ4VzTnfrM1VPi6DPdJtHwtkpPXaRcxAaWrv3QQz7fCkHaoIsdO4hp9xC7n8 LOeHFKvsZj2+e3sBJstZrDFOl/NU41Y+3ES10W59gnGnJBiZrwPHfGvNe+em0YbtLfrf k7OVcnUoy2R0X6KOVxEzE0G2eZe3Duod0Ur7M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728363754; x=1728968554; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Pxi2ug8hQ1leGmc2gZxWeO2a5Yus7SNrn9aUqgKLhvY=; b=bO/0PVLixtzJmi71Pj+duQ+JWY3PTPv/BssnjVU8bye4kXjPuAFVmL4Zzj89UVIa8d rOWgkDyS4bd1YExjrJL6hcE/MpMJ0iSYi1Dgrf1JgFxHPRFWtEXicm/LtT+JFHkqkepD nGC2uu/Bt5YZLoCWWFSDsFbAK5nLoFAyeMN82Jxj7ILE85AjadnvtosxAAaZK7WUe7BQ 9IrRCFQXdzwoC4Y1OJLZwaQoHQq2h3l3Dx+GS0MxW1ZDH0T31WGYVqVpBymgUUe9+jl9 M5gMfKQWQUJiXIBvSVwJq9loK+4CkKON/wGgZ+56uU6wxLcA/TTFkx/qY/tLlL5Kke/s 9Cjg== X-Forwarded-Encrypted: i=1; AJvYcCUI83GIUc0+JCKd8uUxzba+UewZkL59n2ASTBa2s5weVuLEFuyACm0KVvU2PjomIjabx8v8YeysgF5xqU8fVYlO@lists.infradead.org X-Gm-Message-State: AOJu0Yz+bP+wNb/r/pE7x1EhZVPp892bu4Y+r/VTPnqbw1v4Wk1SAOa0 SefFDUeE+lJp2YCg9fvyfobSITug788YpeiHJbql6CDhgYUVAl1XCm4fBFpgmw== X-Google-Smtp-Source: AGHT+IHQAo+KmmP+EUY6IKV7G7h/MEpeQjlzyOoQJ0+TpzBAm4RCGeQJG3vbYMnnym1G9bLx2tjQEA== X-Received: by 2002:a17:902:e547:b0:20b:7be8:8eb9 with SMTP id d9443c01a7336-20bff04dbf4mr186119895ad.54.1728363753761; Mon, 07 Oct 2024 22:02:33 -0700 (PDT) Received: from [192.168.68.79] ([136.52.15.143]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20c54010718sm4133905ad.240.2024.10.07.22.02.32 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 07 Oct 2024 22:02:33 -0700 (PDT) Message-ID: Date: Mon, 7 Oct 2024 22:02:32 -0700 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] firmware: arm_scmi: Queue in scmi layer for mailbox implementation To: Peng Fan , Cristian Marussi , Sudeep Holla Cc: "arm-scmi@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "bcm-kernel-feedback-list@broadcom.com" , "florian.fainelli@broadcom.com" References: <20241004221257.2888603-1-justin.chen@broadcom.com> <1ad5c4e9-9f98-40ab-afa4-a7939781e8cc@broadcom.com> Content-Language: en-US From: Justin Chen In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241007_220235_610453_CDD60429 X-CRM114-Status: GOOD ( 22.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 10/7/2024 7:43 PM, Peng Fan wrote: >> Subject: Re: [PATCH] firmware: arm_scmi: Queue in scmi layer for >> mailbox implementation >> >> >> >> On 10/7/24 6:10 AM, Cristian Marussi wrote: >>> On Mon, Oct 07, 2024 at 02:04:10PM +0100, Sudeep Holla wrote: >>>> On Fri, Oct 04, 2024 at 03:12:57PM -0700, Justin Chen wrote: >>>>> The mailbox layer has its own queue. However this confuses the >> per >>>>> message timeouts since the clock starts ticking the moment the >>>>> messages get queued up. So all messages in the queue have there >>>>> timeout clocks ticking instead of only the message inflight. To fix >>>>> this, lets move the queue back into the SCMI layer. >>>>> >>>> >>>> I think this has come up in the past. We have avoided adding >> addition >>>> locking here as the mailbox layer takes care of it. Has anything >>>> changed recently ? >>> >>> I asked for an explanation in my reply (we crossed each other mails >>> probably) since it alredy came up in the past a few times and central >>> locking seemed not to be needed...here the difference is about the >>> reason...Justin talks about message timeouts related to the queueing >>> process..so I asked to better explain the detail (and the anbomaly >>> observed) since it still does not seem to me that even in this case >>> the lock is needed....anyway I can definitely be woring of course :D >>> >> >> Hello Cristian, >> >> Thanks for the response. I'll try to elaborate. >> >> When comparing SMC and mailbox transport, we noticed mailbox >> transport timesout much quicker when under load. Originally we >> thought this was the latency of the mailbox implementation, but after >> debugging we noticed a weird behavior. We saw SMCI transactions >> timing out before the mailbox even transmitted the message. >> >> This issue lies in the SCMI layer. drivers/firmware/arm_scmi/driver.c >> do_xfer() function. >> >> The fundamental issue is send_message() blocks for SMC transport, but >> doesn't block for mailbox transport. So if send_message() doesn't block >> we can have multiple messages waiting at >> scmi_wait_for_message_response(). >> >> SMC looks like this >> CPU #0 SCMI message 0 -> calls send_message() then calls >> scmi_wait_for_message_response(), timesout after 30ms. >> CPU #1 SCMI message 1 -> blocks at send_message() waiting for SCMI >> message 0 to complete. >> >> Mailbox looks like this >> CPU #0 SCMI message 0 -> calls send_message(), mailbox layer queues >> up message, mailbox layer sees no message is outgoing and sends it. >> CPU waits at scmi_wait_for_message_response(), timesout after 30ms >> CPU #1 SCMI message 1 -> calls send_message(), mailbox layer queues >> up message, mailbox layer sees message pending, hold message in >> queue. CPU waits at scmi_wait_for_message_response(), timesout after >> 30ms. >> >> Lets say if transport takes 25ms. The first message would succeed, the >> second message would timeout after 5ms. > > Each xfer has its own completion, how the 2nd impacts the 1st? > > Regards, > Peng. > Hello Peng, The mailbox layer queues messages and doesn't block when send_message() is called. So we can have both xfer waiting for completion at the same time. Lets assume a message takes 25ms to complete, with a 30ms timeout. 0ms Message #0 is queued in mailbox layer and sent out, then sits at scmi_wait_for_message_response() with a timeout of 30ms 1ms Message #1 is queued in mailbox layer but not sent out yet. Since send_message() doesn't block, it also sits at scmi_wait_for_message_response() with a timeout of 30ms ... 25ms Message #0 is completed, txdone is called and Message #1 is sent out 31ms Message #1 times out since the count started at 1ms. Even though it has only been inflight for 6ms. Thanks, Justin >> >> Hopefully this makes sense. >> >> Justin >> >