[PATCH] firmware: imx: scu: Ensure sequential TX

[PATCH] firmware: imx: scu: Ensure sequential TX

[Leonard Crestez]

SCU requires that all messages words are written sequentially but linux MU
driver implements multiple independent channels for each register so ordering
between different channels must be ensured by SCU API interface.

Wait for tx_done before every send to ensure that no queueing happens at the
mailbox channel level.

Fixes: edbee095fafb ("firmware: imx: add SCU firmware driver support")
Signed-off-by: Leonard Crestez <leonard.crestez@nxp.com>
Cc: <stable@vger.kernel.org>
---
 drivers/firmware/imx/imx-scu.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

This manifests as "SCU timeout" message followed by system hang.

This is not a very pretty fix but avoids inserting additional waits
except in extremely rare circumstances.

An alternative would be to implement a new type of mailbox channel which
handles all 4 registers together. Exposing the MU as 4 independent
channels is very awkward.

diff --git a/drivers/firmware/imx/imx-scu.c b/drivers/firmware/imx/imx-scu.c
index 03b43b7a6d1d..f71eaa5bf52d 100644
--- a/drivers/firmware/imx/imx-scu.c
+++ b/drivers/firmware/imx/imx-scu.c
@@ -27,10 +27,11 @@ struct imx_sc_chan {
 	struct imx_sc_ipc *sc_ipc;
 
 	struct mbox_client cl;
 	struct mbox_chan *ch;
 	int idx;
+	struct completion tx_done;
 };
 
 struct imx_sc_ipc {
 	/* SCU uses 4 Tx and 4 Rx channels */
 	struct imx_sc_chan chans[SCU_MU_CHAN_NUM];
@@ -98,10 +99,18 @@ int imx_scu_get_handle(struct imx_sc_ipc **ipc)
 	*ipc = imx_sc_ipc_handle;
 	return 0;
 }
 EXPORT_SYMBOL(imx_scu_get_handle);
 
+/* Callback called when the word of a message is ack-ed, eg read by SCU */
+static void imx_scu_tx_done(struct mbox_client *cl, void *mssg, int r)
+{
+	struct imx_sc_chan *sc_chan = container_of(cl, struct imx_sc_chan, cl);
+
+	complete(&sc_chan->tx_done);
+}
+
 static void imx_scu_rx_callback(struct mbox_client *c, void *msg)
 {
 	struct imx_sc_chan *sc_chan = container_of(c, struct imx_sc_chan, cl);
 	struct imx_sc_ipc *sc_ipc = sc_chan->sc_ipc;
 	struct imx_sc_rpc_msg *hdr;
@@ -147,10 +156,23 @@ static int imx_scu_ipc_write(struct imx_sc_ipc *sc_ipc, void *msg)
 	dev_dbg(sc_ipc->dev, "RPC SVC %u FUNC %u SIZE %u\n", hdr->svc,
 		hdr->func, hdr->size);
 
 	for (i = 0; i < hdr->size; i++) {
 		sc_chan = &sc_ipc->chans[i % 4];
+
+		/*
+		 * SCU requires that all messages words are written
+		 * sequentially but linux MU driver implements multiple
+		 * independent channels for each register so ordering between
+		 * different channels must be ensured by SCU API interface.
+		 *
+		 * Wait for tx_done before every send to ensure that no
+		 * queueing happens at the mailbox channel level.
+		 */
+		wait_for_completion(&sc_chan->tx_done);
+		reinit_completion(&sc_chan->tx_done);
+
 		ret = mbox_send_message(sc_chan->ch, &data[i]);
 		if (ret < 0)
 			return ret;
 	}
 
@@ -245,10 +267,15 @@ static int imx_scu_probe(struct platform_device *pdev)
 		cl->dev = dev;
 		cl->tx_block = false;
 		cl->knows_txdone = true;
 		cl->rx_callback = imx_scu_rx_callback;
 
+		/* Initial tx_done completion as "done" */
+		cl->tx_done = imx_scu_tx_done;
+		init_completion(&sc_chan->tx_done);
+		complete(&sc_chan->tx_done);
+
 		sc_chan->sc_ipc = sc_ipc;
 		sc_chan->idx = i % 4;
 		sc_chan->ch = mbox_request_channel_byname(cl, chan_name);
 		if (IS_ERR(sc_chan->ch)) {
 			ret = PTR_ERR(sc_chan->ch);
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [PATCH] firmware: imx: scu: Ensure sequential TX

[Peng Fan]

> Subject: [PATCH] firmware: imx: scu: Ensure sequential TX
> 
> SCU requires that all messages words are written sequentially but linux MU
> driver implements multiple independent channels for each register so
> ordering between different channels must be ensured by SCU API interface.
> 
> Wait for tx_done before every send to ensure that no queueing happens at
> the mailbox channel level.
> 
> Fixes: edbee095fafb ("firmware: imx: add SCU firmware driver support")
> Signed-off-by: Leonard Crestez <leonard.crestez@nxp.com>
> Cc: <stable@vger.kernel.org>

I am working on binding 4 registers in one channel per MU chapter using
example. But since this is a critical fix,

Reviewed-by: Peng Fan <peng.fan@nxp.com>

> ---
>  drivers/firmware/imx/imx-scu.c | 27 +++++++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
> 
> This manifests as "SCU timeout" message followed by system hang.
> 
> This is not a very pretty fix but avoids inserting additional waits except in
> extremely rare circumstances.
> 
> An alternative would be to implement a new type of mailbox channel which
> handles all 4 registers together. Exposing the MU as 4 independent channels
> is very awkward.
> 
> diff --git a/drivers/firmware/imx/imx-scu.c b/drivers/firmware/imx/imx-scu.c
> index 03b43b7a6d1d..f71eaa5bf52d 100644
> --- a/drivers/firmware/imx/imx-scu.c
> +++ b/drivers/firmware/imx/imx-scu.c
> @@ -27,10 +27,11 @@ struct imx_sc_chan {
>  	struct imx_sc_ipc *sc_ipc;
> 
>  	struct mbox_client cl;
>  	struct mbox_chan *ch;
>  	int idx;
> +	struct completion tx_done;
>  };
> 
>  struct imx_sc_ipc {
>  	/* SCU uses 4 Tx and 4 Rx channels */
>  	struct imx_sc_chan chans[SCU_MU_CHAN_NUM]; @@ -98,10 +99,18
> @@ int imx_scu_get_handle(struct imx_sc_ipc **ipc)
>  	*ipc = imx_sc_ipc_handle;
>  	return 0;
>  }
>  EXPORT_SYMBOL(imx_scu_get_handle);
> 
> +/* Callback called when the word of a message is ack-ed, eg read by SCU
> +*/ static void imx_scu_tx_done(struct mbox_client *cl, void *mssg, int
> +r) {
> +	struct imx_sc_chan *sc_chan = container_of(cl, struct imx_sc_chan,
> +cl);
> +
> +	complete(&sc_chan->tx_done);
> +}
> +
>  static void imx_scu_rx_callback(struct mbox_client *c, void *msg)  {
>  	struct imx_sc_chan *sc_chan = container_of(c, struct imx_sc_chan, cl);
>  	struct imx_sc_ipc *sc_ipc = sc_chan->sc_ipc;
>  	struct imx_sc_rpc_msg *hdr;
> @@ -147,10 +156,23 @@ static int imx_scu_ipc_write(struct imx_sc_ipc
> *sc_ipc, void *msg)
>  	dev_dbg(sc_ipc->dev, "RPC SVC %u FUNC %u SIZE %u\n", hdr->svc,
>  		hdr->func, hdr->size);
> 
>  	for (i = 0; i < hdr->size; i++) {
>  		sc_chan = &sc_ipc->chans[i % 4];
> +
> +		/*
> +		 * SCU requires that all messages words are written
> +		 * sequentially but linux MU driver implements multiple
> +		 * independent channels for each register so ordering between
> +		 * different channels must be ensured by SCU API interface.
> +		 *
> +		 * Wait for tx_done before every send to ensure that no
> +		 * queueing happens at the mailbox channel level.
> +		 */
> +		wait_for_completion(&sc_chan->tx_done);
> +		reinit_completion(&sc_chan->tx_done);
> +
>  		ret = mbox_send_message(sc_chan->ch, &data[i]);
>  		if (ret < 0)
>  			return ret;
>  	}
> 
> @@ -245,10 +267,15 @@ static int imx_scu_probe(struct platform_device
> *pdev)
>  		cl->dev = dev;
>  		cl->tx_block = false;
>  		cl->knows_txdone = true;
>  		cl->rx_callback = imx_scu_rx_callback;
> 
> +		/* Initial tx_done completion as "done" */
> +		cl->tx_done = imx_scu_tx_done;
> +		init_completion(&sc_chan->tx_done);
> +		complete(&sc_chan->tx_done);
> +
>  		sc_chan->sc_ipc = sc_ipc;
>  		sc_chan->idx = i % 4;
>  		sc_chan->ch = mbox_request_channel_byname(cl, chan_name);
>  		if (IS_ERR(sc_chan->ch)) {
>  			ret = PTR_ERR(sc_chan->ch);
> --
> 2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] firmware: imx: scu: Ensure sequential TX

[Oleksij Rempel]

[-- Attachment #1.1: Type: text/plain, Size: 4230 bytes --]

On Thu, Feb 20, 2020 at 06:10:01PM +0200, Leonard Crestez wrote:
> SCU requires that all messages words are written sequentially but linux MU
> driver implements multiple independent channels for each register so ordering
> between different channels must be ensured by SCU API interface.
> 
> Wait for tx_done before every send to ensure that no queueing happens at the
> mailbox channel level.
> 
> Fixes: edbee095fafb ("firmware: imx: add SCU firmware driver support")
> Signed-off-by: Leonard Crestez <leonard.crestez@nxp.com>
> Cc: <stable@vger.kernel.org>

Did you measured performance regression with this change? It will be
good to have a note about it in the commit message.

Reviewed-by:: Oleksij Rempel <o.rempel@pengutronix.de>

> ---
>  drivers/firmware/imx/imx-scu.c | 27 +++++++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
> 
> This manifests as "SCU timeout" message followed by system hang.
> 
> This is not a very pretty fix but avoids inserting additional waits
> except in extremely rare circumstances.
> 
> An alternative would be to implement a new type of mailbox channel which
> handles all 4 registers together. Exposing the MU as 4 independent
> channels is very awkward.
> 
> diff --git a/drivers/firmware/imx/imx-scu.c b/drivers/firmware/imx/imx-scu.c
> index 03b43b7a6d1d..f71eaa5bf52d 100644
> --- a/drivers/firmware/imx/imx-scu.c
> +++ b/drivers/firmware/imx/imx-scu.c
> @@ -27,10 +27,11 @@ struct imx_sc_chan {
>  	struct imx_sc_ipc *sc_ipc;
>  
>  	struct mbox_client cl;
>  	struct mbox_chan *ch;
>  	int idx;
> +	struct completion tx_done;
>  };
>  
>  struct imx_sc_ipc {
>  	/* SCU uses 4 Tx and 4 Rx channels */
>  	struct imx_sc_chan chans[SCU_MU_CHAN_NUM];
> @@ -98,10 +99,18 @@ int imx_scu_get_handle(struct imx_sc_ipc **ipc)
>  	*ipc = imx_sc_ipc_handle;
>  	return 0;
>  }
>  EXPORT_SYMBOL(imx_scu_get_handle);
>  
> +/* Callback called when the word of a message is ack-ed, eg read by SCU */
> +static void imx_scu_tx_done(struct mbox_client *cl, void *mssg, int r)
> +{
> +	struct imx_sc_chan *sc_chan = container_of(cl, struct imx_sc_chan, cl);
> +
> +	complete(&sc_chan->tx_done);
> +}
> +
>  static void imx_scu_rx_callback(struct mbox_client *c, void *msg)
>  {
>  	struct imx_sc_chan *sc_chan = container_of(c, struct imx_sc_chan, cl);
>  	struct imx_sc_ipc *sc_ipc = sc_chan->sc_ipc;
>  	struct imx_sc_rpc_msg *hdr;
> @@ -147,10 +156,23 @@ static int imx_scu_ipc_write(struct imx_sc_ipc *sc_ipc, void *msg)
>  	dev_dbg(sc_ipc->dev, "RPC SVC %u FUNC %u SIZE %u\n", hdr->svc,
>  		hdr->func, hdr->size);
>  
>  	for (i = 0; i < hdr->size; i++) {
>  		sc_chan = &sc_ipc->chans[i % 4];
> +
> +		/*
> +		 * SCU requires that all messages words are written
> +		 * sequentially but linux MU driver implements multiple
> +		 * independent channels for each register so ordering between
> +		 * different channels must be ensured by SCU API interface.
> +		 *
> +		 * Wait for tx_done before every send to ensure that no
> +		 * queueing happens at the mailbox channel level.
> +		 */
> +		wait_for_completion(&sc_chan->tx_done);
> +		reinit_completion(&sc_chan->tx_done);
> +
>  		ret = mbox_send_message(sc_chan->ch, &data[i]);
>  		if (ret < 0)
>  			return ret;
>  	}
>  
> @@ -245,10 +267,15 @@ static int imx_scu_probe(struct platform_device *pdev)
>  		cl->dev = dev;
>  		cl->tx_block = false;
>  		cl->knows_txdone = true;
>  		cl->rx_callback = imx_scu_rx_callback;
>  
> +		/* Initial tx_done completion as "done" */
> +		cl->tx_done = imx_scu_tx_done;
> +		init_completion(&sc_chan->tx_done);
> +		complete(&sc_chan->tx_done);
> +
>  		sc_chan->sc_ipc = sc_ipc;
>  		sc_chan->idx = i % 4;
>  		sc_chan->ch = mbox_request_channel_byname(cl, chan_name);
>  		if (IS_ERR(sc_chan->ch)) {
>  			ret = PTR_ERR(sc_chan->ch);
> -- 
> 2.17.1
> 
> 
> 

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] firmware: imx: scu: Ensure sequential TX

[Leonard Crestez]

On 21.02.2020 07:31, Oleksij Rempel wrote:
> On Thu, Feb 20, 2020 at 06:10:01PM +0200, Leonard Crestez wrote:
>> SCU requires that all messages words are written sequentially but linux MU
>> driver implements multiple independent channels for each register so ordering
>> between different channels must be ensured by SCU API interface.
>>
>> Wait for tx_done before every send to ensure that no queueing happens at the
>> mailbox channel level.
>>
>> Fixes: edbee095fafb ("firmware: imx: add SCU firmware driver support")
>> Signed-off-by: Leonard Crestez <leonard.crestez@nxp.com>
>> Cc: <stable@vger.kernel.org>
> 
> Did you measured performance regression with this change? It will be
> good to have a note about it in the commit message.

I tried to measure "boot time" but measurement was inconclusive, impact 
is too small and gets lots inside stuff like ethernet phy setup for nfs 
root.

I wrote a special stress test module doing simple calls to 
IMX_SC_MISC_FUNC_BUILD_INFO and IMX_SC_RM_FUNC_FIND_MEMREG.

  * with this patch: ~68us/iteration
  * with this patch: ~62us/iteration, eventual SCU timeout
  * with tx_block=true: ~115us/iteration
  * with imx_4.14.y: ~42us/iteration

Source here (some tweaking required):
https://github.com/cdleonard/imx-scu-test/blob/master/imx-scu-test.c

Improved performance on imx_4.14.y is likely because no TX irqs are 
enabled since sender doesn't actually care.

>> ---
>>   drivers/firmware/imx/imx-scu.c | 27 +++++++++++++++++++++++++++
>>   1 file changed, 27 insertions(+)
>>
>> This manifests as "SCU timeout" message followed by system hang.
>>
>> This is not a very pretty fix but avoids inserting additional waits
>> except in extremely rare circumstances.
>>
>> An alternative would be to implement a new type of mailbox channel which
>> handles all 4 registers together. Exposing the MU as 4 independent
>> channels is very awkward.
>>
>> diff --git a/drivers/firmware/imx/imx-scu.c b/drivers/firmware/imx/imx-scu.c
>> index 03b43b7a6d1d..f71eaa5bf52d 100644
>> --- a/drivers/firmware/imx/imx-scu.c
>> +++ b/drivers/firmware/imx/imx-scu.c
>> @@ -27,10 +27,11 @@ struct imx_sc_chan {
>>   	struct imx_sc_ipc *sc_ipc;
>>   
>>   	struct mbox_client cl;
>>   	struct mbox_chan *ch;
>>   	int idx;
>> +	struct completion tx_done;
>>   };
>>   
>>   struct imx_sc_ipc {
>>   	/* SCU uses 4 Tx and 4 Rx channels */
>>   	struct imx_sc_chan chans[SCU_MU_CHAN_NUM];
>> @@ -98,10 +99,18 @@ int imx_scu_get_handle(struct imx_sc_ipc **ipc)
>>   	*ipc = imx_sc_ipc_handle;
>>   	return 0;
>>   }
>>   EXPORT_SYMBOL(imx_scu_get_handle);
>>   
>> +/* Callback called when the word of a message is ack-ed, eg read by SCU */
>> +static void imx_scu_tx_done(struct mbox_client *cl, void *mssg, int r)
>> +{
>> +	struct imx_sc_chan *sc_chan = container_of(cl, struct imx_sc_chan, cl);
>> +
>> +	complete(&sc_chan->tx_done);
>> +}
>> +
>>   static void imx_scu_rx_callback(struct mbox_client *c, void *msg)
>>   {
>>   	struct imx_sc_chan *sc_chan = container_of(c, struct imx_sc_chan, cl);
>>   	struct imx_sc_ipc *sc_ipc = sc_chan->sc_ipc;
>>   	struct imx_sc_rpc_msg *hdr;
>> @@ -147,10 +156,23 @@ static int imx_scu_ipc_write(struct imx_sc_ipc *sc_ipc, void *msg)
>>   	dev_dbg(sc_ipc->dev, "RPC SVC %u FUNC %u SIZE %u\n", hdr->svc,
>>   		hdr->func, hdr->size);
>>   
>>   	for (i = 0; i < hdr->size; i++) {
>>   		sc_chan = &sc_ipc->chans[i % 4];
>> +
>> +		/*
>> +		 * SCU requires that all messages words are written
>> +		 * sequentially but linux MU driver implements multiple
>> +		 * independent channels for each register so ordering between
>> +		 * different channels must be ensured by SCU API interface.
>> +		 *
>> +		 * Wait for tx_done before every send to ensure that no
>> +		 * queueing happens at the mailbox channel level.
>> +		 */
>> +		wait_for_completion(&sc_chan->tx_done);
>> +		reinit_completion(&sc_chan->tx_done);
>> +
>>   		ret = mbox_send_message(sc_chan->ch, &data[i]);
>>   		if (ret < 0)
>>   			return ret;
>>   	}
>>   
>> @@ -245,10 +267,15 @@ static int imx_scu_probe(struct platform_device *pdev)
>>   		cl->dev = dev;
>>   		cl->tx_block = false;
>>   		cl->knows_txdone = true;
>>   		cl->rx_callback = imx_scu_rx_callback;
>>   
>> +		/* Initial tx_done completion as "done" */
>> +		cl->tx_done = imx_scu_tx_done;
>> +		init_completion(&sc_chan->tx_done);
>> +		complete(&sc_chan->tx_done);
>> +
>>   		sc_chan->sc_ipc = sc_ipc;
>>   		sc_chan->idx = i % 4;
>>   		sc_chan->ch = mbox_request_channel_byname(cl, chan_name);
>>   		if (IS_ERR(sc_chan->ch)) {
>>   			ret = PTR_ERR(sc_chan->ch);
>> -- 
>> 2.17.1
>>
>>
>>
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] firmware: imx: scu: Ensure sequential TX

[Shawn Guo]

On Thu, Feb 20, 2020 at 06:10:01PM +0200, Leonard Crestez wrote:
> SCU requires that all messages words are written sequentially but linux MU
> driver implements multiple independent channels for each register so ordering
> between different channels must be ensured by SCU API interface.
> 
> Wait for tx_done before every send to ensure that no queueing happens at the
> mailbox channel level.
> 
> Fixes: edbee095fafb ("firmware: imx: add SCU firmware driver support")
> Signed-off-by: Leonard Crestez <leonard.crestez@nxp.com>
> Cc: <stable@vger.kernel.org>

Applied, thanks.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel