From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F086C61DA3 for ; Fri, 24 Feb 2023 07:49:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229484AbjBXHth (ORCPT ); Fri, 24 Feb 2023 02:49:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229452AbjBXHtg (ORCPT ); Fri, 24 Feb 2023 02:49:36 -0500 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 97A624DE38 for ; Thu, 23 Feb 2023 23:49:34 -0800 (PST) Received: by mail-wr1-x429.google.com with SMTP id bv17so140857wrb.5 for ; Thu, 23 Feb 2023 23:49:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=QbxEvBcANfCyv04vVI+P20GroH/CL1OZ8pzexawzCPA=; b=tnQWyJSxP15q1apDQgYJZl7TiYEzVSdSDue1GfVVOaD0fB7R0X/Bpt/P+s54EvMarj Y1GC+LfEvoa4d+LHbYhq1lLahNstbjvLM3CPlLNdcTMWd+6fEpU5pCFUdd1VfeuNgKwk ibvjVqYpxqAiwq34+4XApexuqjYGbp4gt7qZw1iuaxGK1TvncfYmapRjGkD85NIMMmmP NbuQi4uWSQWuuy9ihIFez+nKuGnq8M3VjfTqgeRzTTrLM/7QimIK9UBuRM43t1GjyADK yey+QqJZp4HrIHUMyIt6xRNdgPgUO0A11h1Oks2iBavOIBt53d+FQQPLO4ynnVsaUe+U aQBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QbxEvBcANfCyv04vVI+P20GroH/CL1OZ8pzexawzCPA=; b=p2IuGlybFXxOjf8MyLWUpvt4KQoWOzeXESLnCnSkeH9Gv4ldMh1lFSR6X/X9O+SSQk GAP0yawicZ/ntEhD0gmDxT56Z9JnmlHPXCMwXDeZJXShReCMfqzG4K3qJe/gSNn09XhO KFzmI21MtU1Q0asFrT47Z00b9t77zcUg6TnAqQX4fVjhxk002mlMlBlZQS4k2LG36Kse mVOTbZ/YorLf+fTodIKb+Xxse4wGzZdkKfVh7i1AG1ZL1/VqlCQ+QwDTvA0ERguSblno WB75ATv5/D1LKYCZNo0iLxBVx3xz6bbDX+7PegUmiXNitWtQY3RTWJl+qQVCAd6mzqRN 8tcA== X-Gm-Message-State: AO0yUKVrTalea3XnvOngXixYkGmOfhdcwtZ57vEV5WGnEdH/doNSJrkq KwvFNo6DjrL3xLAR3wjDyvKUrQ== X-Google-Smtp-Source: AK7set/nHyhaRUq0Yd+9Qdd+3r2tbNOhcXsgs9HAHAccmP/VT5YKMYKugqE9ITyi4t9jCFYpx+UWhg== X-Received: by 2002:a05:6000:98c:b0:2c5:5ee9:6b27 with SMTP id by12-20020a056000098c00b002c55ee96b27mr11889818wrb.13.1677224973014; Thu, 23 Feb 2023 23:49:33 -0800 (PST) Received: from [192.168.1.195] ([5.133.47.210]) by smtp.googlemail.com with ESMTPSA id j21-20020a5d6e55000000b002c3f81c51b6sm13600444wrz.90.2023.02.23.23.49.31 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 23 Feb 2023 23:49:32 -0800 (PST) Message-ID: <78fc83d7-a31b-c6bd-4e08-f0696e0a275b@linaro.org> Date: Fri, 24 Feb 2023 07:49:31 +0000 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: [PATCH v10 07/26] mailbox: Add Gunyah message queue mailbox Content-Language: en-US To: Elliot Berman , Alex Elder , Prakruthi Deepak Heragu , Jonathan Corbet , Jassi Brar Cc: Murali Nalajala , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Dmitry Baryshkov , Bjorn Andersson , Konrad Dybcio , Arnd Bergmann , Greg Kroah-Hartman , Rob Herring , Krzysztof Kozlowski , Bagas Sanjaya , Catalin Marinas , linux-arm-msm@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org References: <20230214211229.3239350-1-quic_eberman@quicinc.com> <20230214212316.3309053-1-quic_eberman@quicinc.com> <576aed85-a566-3645-559e-06b2135cf57f@quicinc.com> <7e3170e4-c530-0b5b-903f-e5ea6d8268dc@linaro.org> From: Srinivas Kandagatla In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-doc@vger.kernel.org On 23/02/2023 23:15, Elliot Berman wrote: > > > On 2/23/2023 2:25 AM, Srinivas Kandagatla wrote: >> >> >> On 23/02/2023 00:15, Elliot Berman wrote: >>> >>> >>> On 2/20/2023 5:59 AM, Srinivas Kandagatla wrote: >>>> >>>> >>>> On 14/02/2023 21:23, Elliot Berman wrote: >>>>> Gunyah message queues are a unidirectional inter-VM pipe for >>>>> messages up >>>>> to 1024 bytes. This driver supports pairing a receiver message >>>>> queue and >>>>> a transmitter message queue to expose a single mailbox channel. >>>>> >>>>> Signed-off-by: Elliot Berman >>>>> --- >>>>>   Documentation/virt/gunyah/message-queue.rst |   8 + >>>>>   drivers/mailbox/Makefile                   |   2 + >>>>>   drivers/mailbox/gunyah-msgq.c               | 214 >>>>> ++++++++++++++++++++ >>>>>   include/linux/gunyah.h                      |  56 +++++ >>>>>   4 files changed, 280 insertions(+) >>>>>   create mode 100644 drivers/mailbox/gunyah-msgq.c >>>>> >>>>> diff --git a/Documentation/virt/gunyah/message-queue.rst >>>>> b/Documentation/virt/gunyah/message-queue.rst >>>>> index 0667b3eb1ff9..082085e981e0 100644 >>>>> --- a/Documentation/virt/gunyah/message-queue.rst >>>>> +++ b/Documentation/virt/gunyah/message-queue.rst >>>>> @@ -59,3 +59,11 @@ vIRQ: two TX message queues will have two vIRQs >>>>> (and two capability IDs). >>>>>         |               |         |                 | >>>>> |               | >>>>>         |               |         |                 | >>>>> |               | >>>>>         +---------------+         +-----------------+ >>>>> +---------------+ >>>>> + >>>>> +Gunyah message queues are exposed as mailboxes. To create the >>>>> mailbox, create >>>>> +a mbox_client and call `gh_msgq_init`. On receipt of the RX_READY >>>>> interrupt, >>>>> +all messages in the RX message queue are read and pushed via the >>>>> `rx_callback` >>>>> +of the registered mbox_client. >>>>> + >>>>> +.. kernel-doc:: drivers/mailbox/gunyah-msgq.c >>>>> +   :identifiers: gh_msgq_init >>>>> diff --git a/drivers/mailbox/Makefile b/drivers/mailbox/Makefile >>>>> index fc9376117111..5f929bb55e9a 100644 >>>>> --- a/drivers/mailbox/Makefile >>>>> +++ b/drivers/mailbox/Makefile >>>>> @@ -55,6 +55,8 @@ obj-$(CONFIG_MTK_CMDQ_MBOX)    += mtk-cmdq-mailbox.o >>>>>   obj-$(CONFIG_ZYNQMP_IPI_MBOX)    += zynqmp-ipi-mailbox.o >>>>> +obj-$(CONFIG_GUNYAH)        += gunyah-msgq.o >>>> >>>> Why are we reusing CONFIG_GUNYAH Kconfig symbol for mailbox, why not >>>> CONFIG_GUNYAH_MBOX? >>>> >>> >>> There was some previous discussion about this: >>> >>> https://lore.kernel.org/all/2a7bb5f2-1286-b661-659a-a5037150eae8@quicinc.com/ >>> >>>>> + >>>>>   obj-$(CONFIG_SUN6I_MSGBOX)    += sun6i-msgbox.o >>>>>   obj-$(CONFIG_SPRD_MBOX)       += sprd-mailbox.o >>>>> diff --git a/drivers/mailbox/gunyah-msgq.c >>>>> b/drivers/mailbox/gunyah-msgq.c >>>>> new file mode 100644 >>>>> index 000000000000..03ffaa30ce9b >>>>> --- /dev/null >>>>> +++ b/drivers/mailbox/gunyah-msgq.c >>>>> @@ -0,0 +1,214 @@ >>>>> +// SPDX-License-Identifier: GPL-2.0-only >>>>> +/* >>>>> + * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All >>>>> rights reserved. >>>>> + */ >>>>> + >>>>> +#include >>>>> +#include >>>>> +#include >>>>> +#include >>>>> +#include >>>>> +#include >>>>> +#include >>>>> +#include >>>> >>>> ... >>>> >>>>> +/* Fired when message queue transitions from "full" to "space >>>>> available" to send messages */ >>>>> +static irqreturn_t gh_msgq_tx_irq_handler(int irq, void *data) >>>>> +{ >>>>> +    struct gh_msgq *msgq = data; >>>>> + >>>>> +    mbox_chan_txdone(gh_msgq_chan(msgq), 0); >>>>> + >>>>> +    return IRQ_HANDLED; >>>>> +} >>>>> + >>>>> +/* Fired after sending message and hypercall told us there was >>>>> more space available. */ >>>>> +static void gh_msgq_txdone_tasklet(struct tasklet_struct *tasklet) >>>> >>>> Tasklets have been long deprecated, consider using workqueues in >>>> this particular case. >>>> >>> >>> Workqueues have higher latency and tasklets came as recommendation >>> from Jassi. drivers/mailbox/imx-mailbox.c uses tasklets in the same way. >>> >>> I did some quick unscientific measurements of ~1000x samples. The >>> median latency for resource manager went from 25.5 us (tasklet) to 26 >>> us (workqueue) (2% slower). The mean went from 28.7 us to 32.5 us >>> (13% slower). Obviously, the outliers for workqueues were much more >>> extreme. >> >> TBH, this is expected because we are only testing resource manager, >> Note   the advantage that you will see shifting from tasket to >> workqueues is on overall system latencies and some drivers performance >> that need to react to events. >> >> please take some time to read this nice article about this >> https://lwn.net/Articles/830964/ >> > > Hmm, this article is from 2020 and there was another effort in 2007. > Neither seems to have succeeded. I'd like to stick to same mechanisms as > other mailbox controllers. I don't want to block this series because of this. We will have more opportunity to improve this once some system wide profiling is done. AFAIU, In this system we will have atleast 2 tasklets between VM and RM and 2 per inter-vm, so if the number of tasklets increase in the system will be potentially spending more time in soft irq handling it. At somepoint in time its good to get some profiling done using bcc/softirqs to see how much time is spent on softirqs. --srini > > Jassi, do you have any preferences? > > Thanks, > Elliot > >