Linux EDAC development
 help / color / mirror / Atom feed
From: "Pandey, Radhey Shyam" <radheys@amd.com>
To: Shubhrajyoti Datta <shubhrajyoti.datta@amd.com>,
	linux-edac@vger.kernel.org
Cc: git@amd.com, shubhrajyoti.datta@gmail.com, michal.simek@amd.com,
	Radhey Shyam Pandey <radhey.shyam.pandey@amd.com>,
	Krzysztof Kozlowski <krzk@kernel.org>,
	Rob Herring <robh@kernel.org>, Conor Dooley <conor+dt@kernel.org>,
	Borislav Petkov <bp@alien8.de>, Tony Luck <tony.luck@intel.com>,
	linux-kernel@vger.kernel.org, devicetree@vger.kernel.org
Subject: Re: [PATCH v2 06/12] EDAC/versalnet: Wait for RPMsg endpoint initialization
Date: Sat, 21 Nov 2026 20:58:37 +0530	[thread overview]
Message-ID: <eb3ea65f-7602-48b5-b983-b09c091a816e@amd.com> (raw)
In-Reply-To: <20261118142213.100581-7-shubhrajyoti.datta@amd.com>

On 11/18/2026 7:50 PM, Shubhrajyoti Datta wrote:
> From: Hayden Rinn <haydenrinn@microsoft.com>
> 
> The rpmsg endpoint can be used before rpmsg_probe() completes, and
> can race with asynchronous rpmsg_remove() teardown (e.g. R5 crash
> or shutdown), both leading to use of an uninitialized or freed
> endpoint.
> 
> Add a completion so mc_probe() waits for the endpoint, and a
> "ready" flag so rpmsg_cb() drops events until mci/adec are
> set up. Add ept_lock to serialize priv->ept and priv->mcdi->ept
> across mcdi_request(), rpmsg_probe(), and rpmsg_remove(), clearing
> them on removal and failing mcdi_request() if the channel is down.
> 
> Fixes: d5fe2fec6c40 ("EDAC: Add a driver for the AMD Versal NET DDR controller")
> Signed-off-by: Hayden Rinn <haydenrinn@microsoft.com>

Why Hayden is missed out in to/cc list?

> Co-developed-by: Shubhrajyoti Datta <shubhrajyoti.datta@amd.com>
> Signed-off-by: Shubhrajyoti Datta <shubhrajyoti.datta@amd.com>
> ---
> 
> Changes in v2:
> Patch addition for the rpmsg endpoint race
> 
>   drivers/edac/versalnet_edac.c | 123 ++++++++++++++++++++++++++++++----
>   1 file changed, 111 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/edac/versalnet_edac.c b/drivers/edac/versalnet_edac.c
> index ea90d3ce3e70..90918293f76d 100644
> --- a/drivers/edac/versalnet_edac.c
> +++ b/drivers/edac/versalnet_edac.c
> @@ -5,8 +5,10 @@
>    */
>   
>   #include <linux/cdx/edac_cdx_pcol.h>
> +#include <linux/completion.h>
>   #include <linux/edac.h>
>   #include <linux/module.h>
> +#include <linux/mutex.h>
>   #include <linux/of_device.h>
>   #include <linux/ras.h>
>   #include <linux/remoteproc.h>
> @@ -146,7 +148,22 @@ struct ecc_status {
>    * @adec:		Address decode registers.
>    * @mci:		Memory controller interface.
>    * @ept:		rpmsg endpoint.
> + * @ept_lock:		Protects @ept and @mcdi->ept against concurrent teardown
> + *			by rpmsg_remove() (e.g. remote processor crash/shutdown)
> + *			while mcdi_request() is sending on them.
> + * @rpmsg_ready:	Completion signaled once rpmsg_probe() has run to
> + *			completion success. @ept is only valid to check
> + *			once this is signaled.
>    * @mcdi:		The mcdi handle.
> + * @dev:		Platform device backing this driver instance. Stays
> + *			valid for as long as mcdi_request() can be called,
> + *			unlike @mcdi->rpdev, which is torn down and cleared
> + *			by rpmsg_remove(). Used for logging when there is no
> + *			live rpmsg channel to hang the message off of.

To big description. Lets trim all this AI generated text.

> + * @ready:		Set once edac priv data is initialised. Published with a
> + *			release store and read with an acquire load so that rpmsg_cb
> + *			never calls handle_error() against a @mci entry that hasn't
> + *			been allocated yet.

Same here.

>    */
>   struct mc_priv {
>   	char message[256];
> @@ -159,7 +176,11 @@ struct mc_priv {
>   	u32 adec[ADEC_MAX];
>   	struct mem_ctl_info *mci[NUM_CONTROLLERS];
>   	struct rpmsg_endpoint *ept;
> +	struct mutex ept_lock; /* Protects endpoint */
> +	struct completion rpmsg_ready;
>   	struct cdx_mcdi *mcdi;
> +	struct device *dev;
> +	bool ready;
>   };
>   
>   /*
> @@ -508,6 +529,12 @@ static void mc_init(struct mem_ctl_info *mci, struct device *dev,
>   
>   #define to_mci(k) container_of(k, struct mem_ctl_info, dev)
>   
> +static struct rpmsg_device_id amd_rpmsg_id_table[] = {
> +	{ .name = EDAC_RPMSG_CHAN_NAME },
> +	{ },
> +};
> +MODULE_DEVICE_TABLE(rpmsg, amd_rpmsg_id_table);
> +
>   static unsigned int mcdi_rpc_timeout(struct cdx_mcdi *cdx, unsigned int cmd)
>   {
>   	return MCDI_RPC_TIMEOUT;
> @@ -517,6 +544,7 @@ static void mcdi_request(struct cdx_mcdi *cdx,
>   			 const struct cdx_dword *hdr, size_t hdr_len,
>   			 const struct cdx_dword *sdu, size_t sdu_len)
>   {
> +	struct mc_priv *mc_priv = (struct mc_priv *)amd_rpmsg_id_table[0].driver_data;
>   	void *send_buf;
>   	int ret;
>   
> @@ -527,6 +555,24 @@ static void mcdi_request(struct cdx_mcdi *cdx,
>   	memcpy(send_buf, hdr, hdr_len);
>   	memcpy(send_buf + hdr_len, sdu, sdu_len);
>   
> +	guard(mutex)(&mc_priv->ept_lock);
> +
> +	/*
> +	 * The rpmsg channel can be torn down asynchronously (e.g. the R5
> +	 * remote processor crashing or being shut down) by rpmsg_remove(),
> +	 * which clears both @cdx->ept and @mc_priv->ept under this same
> +	 * lock. Bail out instead of sending on a destroyed endpoint.
> +	 *
> +	 * @cdx->rpdev is cleared by rpmsg_remove() together with @cdx->ept,
> +	 * so it cannot be used for logging here; use the stable platform
> +	 * device instead.
> +	 */
> +	if (!cdx->ept) {
> +		dev_err(mc_priv->dev, "Dropping rpmsg send, channel is down\n");
> +		kfree(send_buf);

Better to do in start and avoid this kfree.

> +		return;
> +	}
> +
>   	ret = rpmsg_send(cdx->ept, send_buf, hdr_len + sdu_len);
>   	if (ret)
>   		dev_err(&cdx->rpdev->dev, "Failed to send rpmsg data: %d\n", ret);
> @@ -601,6 +647,18 @@ static int rpmsg_cb(struct rpmsg_device *rpdev, void *data,
>   		cdx_mcdi_process_cmd(mc_priv->mcdi, (struct cdx_dword *)data, len);
>   		return 0;
>   	}
> +	/*
> +	 * The rpmsg channel (and thus this callback) can become live
> +	 * before init_versalnet() has allocated priv->mci[] and
> +	 * populated priv->adec[]. Drop reports that race ahead
> +	 * of it instead of calling handle_error() against incomplete
> +	 * state.
> +	 */
> +	if (!smp_load_acquire(&mc_priv->ready)) {
> +		dev_warn_ratelimited(&rpdev->dev,
> +				     "Dropping event, driver not ready yet\n");
> +		return 0;
> +	}
>   
>   	sec_sev = result[ERROR_LEVEL];
>   	error_id = result[ERROR_ID];
> @@ -721,12 +779,6 @@ static int rpmsg_cb(struct rpmsg_device *rpdev, void *data,
>   	return 0;
>   }
>   
> -static struct rpmsg_device_id amd_rpmsg_id_table[] = {
> -	{ .name = EDAC_RPMSG_CHAN_NAME },
> -	{ },
> -};
> -MODULE_DEVICE_TABLE(rpmsg, amd_rpmsg_id_table);
> -
>   static int rpmsg_probe(struct rpmsg_device *rpdev)
>   {
>   	struct rpmsg_channel_info chinfo = {
> @@ -734,25 +786,56 @@ static int rpmsg_probe(struct rpmsg_device *rpdev)
>   		.dst = rpdev->dst,
>   		.name = EDAC_RPMSG_CHAN_NAME,
>   	};
> +	struct rpmsg_endpoint *ept;
>   	struct mc_priv *pg;
>   
>   	pg = (struct mc_priv *)amd_rpmsg_id_table[0].driver_data;
>   
> -	pg->ept = rpmsg_create_ept(rpdev, rpmsg_cb, NULL, chinfo);
> -	if (!pg->ept)
> -		return dev_err_probe(&rpdev->dev, -ENXIO, "Failed to create ept for channel %s\n",
> -				     chinfo.name);
> -
>   	dev_set_drvdata(&rpdev->dev, pg);
>   
> +	ept = rpmsg_create_ept(rpdev, rpmsg_cb, NULL, chinfo);
> +	if (!ept) {
> +		return dev_err_probe(&rpdev->dev, -ENXIO,
> +				      "Failed to create ept for channel %s\n",
> +				      chinfo.name);
> +	}
> +
> +	/*
> +	 * Link the new endpoint in under the lock. This also re-syncs
> +	 * mcdi->ept on reconnection after the remote processor recovers
> +	 * from a crash/shutdown, which previously only happened once,
> +	 * during the initial mc_probe().
> +	 */
> +	scoped_guard(mutex, &pg->ept_lock) {
> +		pg->ept = ept;
> +		pg->mcdi->ept = ept;
> +		pg->mcdi->rpdev = rpdev;
> +	}

[1] Critical: rpmsg_probe() dereferences priv->mcdi before mc_probe()
     assigns it.

priv->mcdi is assigned inside setup_mcdi(), which mc_probe() calls
after register_rpmsg_driver(). If the channel binds in that window -
only a kzalloc_obj() and cdx_mcdi_init() wide, but the NS callback
runs in an unrelated context - pg->mcdi is NULL here.

Pre-patch rpmsg_probe() only touched pg->ept, which lives in the
devm-allocated priv, so this is newly introduced.

There is a second, wider instance: if setup_mcdi() fails in
cdx_mcdi_init() it returns with mc_priv->mcdi still NULL, and
"goto err_unreg" runs unregister_rpmsg_driver() -> rpmsg_remove(),
which does mc_priv->mcdi->ept = NULL on NULL.



> +
> +	complete_all(&pg->rpmsg_ready);
> +
>   	return 0;
>   }
>   
>   static void rpmsg_remove(struct rpmsg_device *rpdev)
>   {
>   	struct mc_priv *mc_priv = dev_get_drvdata(&rpdev->dev);
> +	struct rpmsg_endpoint *ept;
> +
> +	/*
> +	 * Clear the shared endpoint pointers before destroying the
> +	 * endpoint, so that mcdi_request() (serialised on the same lock)
> +	 * cannot observe or send on a stale/freed endpoint if the remote
> +	 * processor goes down concurrently.
> +	 */
> +	scoped_guard(mutex, &mc_priv->ept_lock) {
> +		ept = mc_priv->ept;
> +		mc_priv->ept = NULL;
> +		mc_priv->mcdi->ept = NULL;
> +		mc_priv->mcdi->rpdev = NULL;


[2] Critical: use-after-free on the probe error path.

 > @@ -745,25 +786,56 @@ static void rpmsg_remove(struct rpmsg_device 
*rpdev)
 > +     scoped_guard(mutex, &mc_priv->ept_lock) {
 > +             ept = mc_priv->ept;
 > +             mc_priv->ept = NULL;
 > +             mc_priv->mcdi->ept = NULL;
 > +             mc_priv->mcdi->rpdev = NULL;
 > +     }

Combined with the unwind in mc_probe(), which is unchanged by this
patch but newly matters because rpmsg_remove() now writes through
priv->mcdi:

         err_init:
                 cdx_mcdi_finish(priv->mcdi);
                 kfree(priv->mcdi);

         err_unreg:
                 unregister_rpmsg_driver(&amd_rpmsg_driver);

err_init frees priv->mcdi and then falls through to
unregister_rpmsg_driver(), which calls rpmsg_remove() on the still
bound channel, which writes to the freed object. Reachable whenever
init_versalnet() fails, since the channel is bound by then.

mc_remove() has the right order (unregister, then free); only the
error path is inverted.


> +	}
>   
> -	rpmsg_destroy_ept(mc_priv->ept);
> +	rpmsg_destroy_ept(ept);
>   	dev_set_drvdata(&rpdev->dev, NULL);
>   }
>   
> @@ -920,6 +1003,9 @@ static int mc_probe(struct platform_device *pdev)
>   		goto err_alloc;
>   	}
>   
> +	priv->dev = &pdev->dev;
> +	mutex_init(&priv->ept_lock);
> +	init_completion(&priv->rpmsg_ready);
>   	amd_rpmsg_id_table[0].driver_data = (kernel_ulong_t)priv;
>   
>   	rc = register_rpmsg_driver(&amd_rpmsg_driver);
> @@ -932,12 +1018,25 @@ static int mc_probe(struct platform_device *pdev)
>   	if (rc)
>   		goto err_unreg;
>   

[3] Critical: the completion is waited on after the endpoint's only
     pre-init user, so the patch does not fix the bug it describes.

The wait lands after setup_mcdi(), but setup_mcdi() is itself the
first and only user of the endpoint during probe:

mcdi_request() takes the new !cdx->ept exit and returns void,
so the CDX core never learns the send failed and waits out
cdx_mcdi_rpc_timeout(), 10 * HZ. get_ddr_config() only copies on
success and setup_mcdi() returns 0 regardless, so adec[] stays zero and
probe continues: init_one_mc() reads FIELD_GET(MC5_BUS_WIDTH_MASK, 0)
== XDDR5_BUS_WIDTH_64 and registers MC0 with fabricated geometry. (The
announcement normally lands during that stall, so in practice only MC0
is affected.)

Pre-patch this at least produced a WARN_ON(!ept) from rpmsg_send(). The
guard is correct, but without moving the wait above the loop it only
silences the symptom.


> +	if (!wait_for_completion_timeout(&priv->rpmsg_ready, MCDI_RPC_TIMEOUT)) {
> +		dev_err(&pdev->dev, "Timed out waiting for RPMsg endpoint\n");
> +		rc = -ENODEV;
> +		goto err_init;
> +	}
> +
>   	priv->mcdi->r5_rproc = rp;
>   
>   	rc = init_versalnet(priv, pdev);
>   	if (rc)
>   		goto err_init;
>   
> +	/*
> +	 * edac is initialised. Publish this with a release store so rpmsg_cb
> +	 * running concurrently on the rpmsg worker thread, only calls
> +	 * handle_error() once it observes @ready via a matching acquire load.
> +	 */
> +	smp_store_release(&priv->ready, true);
> +
>   	return 0;
>   
>   err_init:


  reply	other threads:[~2026-11-21 15:29 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 14:20 [PATCH v2 00/12] EDAC/versalnet: Fix error handling, teardown, and robustness Shubhrajyoti Datta
2026-11-18 14:20 ` [PATCH v2 01/12] EDAC/versalnet: Fix device_register() error handling in init_one_mc() Shubhrajyoti Datta
2026-11-20 13:38   ` Pandey, Radhey Shyam
2026-11-18 14:20 ` [PATCH v2 02/12] EDAC/versalnet: Use designated initializer for rpmsg_channel_info Shubhrajyoti Datta
2026-11-20 13:39   ` Pandey, Radhey Shyam
2026-11-18 14:20 ` [PATCH v2 03/12] EDAC/versalnet: Fix use-after-free in remove_one_mc() Shubhrajyoti Datta
2026-11-18 14:20 ` [PATCH v2 04/12] EDAC/versalnet: Add bounds validation in rpmsg_cb() Shubhrajyoti Datta
2026-11-20 15:43   ` Pandey, Radhey Shyam
2026-11-18 14:20 ` [PATCH v2 05/12] EDAC/versalnet: Make dwidth per-controller Shubhrajyoti Datta
2026-11-20 16:54   ` Pandey, Radhey Shyam
2026-11-18 14:20 ` [PATCH v2 06/12] EDAC/versalnet: Wait for RPMsg endpoint initialization Shubhrajyoti Datta
2026-11-21 15:28   ` Pandey, Radhey Shyam [this message]
2026-11-18 14:20 ` [PATCH v2 07/12] EDAC/versalnet: Initialize MCDI before RPMsg registration Shubhrajyoti Datta
2026-11-21 15:33   ` Pandey, Radhey Shyam
2026-11-18 14:20 ` [PATCH v2 08/12] EDAC/versalnet: Guard against NULL drvdata in rpmsg_cb() Shubhrajyoti Datta
2026-11-18 14:20 ` [PATCH v2 09/12] EDAC/versalnet: Move platform_set_drvdata() to mc_probe() Shubhrajyoti Datta
2026-11-18 14:20 ` [PATCH v2 10/12] dt-bindings: memory: xlnx,versal-net-ddrmc5: Add amd,num-ctrl property Shubhrajyoti Datta
2026-11-18 14:20 ` [PATCH v2 11/12] EDAC/versalnet: use device tree property for number of controllers Shubhrajyoti Datta
2026-11-18 14:20 ` [PATCH v2 12/12] EDAC/versalnet: Make get_ddr_config() return errors Shubhrajyoti Datta

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=eb3ea65f-7602-48b5-b983-b09c091a816e@amd.com \
    --to=radheys@amd.com \
    --cc=bp@alien8.de \
    --cc=conor+dt@kernel.org \
    --cc=devicetree@vger.kernel.org \
    --cc=git@amd.com \
    --cc=krzk@kernel.org \
    --cc=linux-edac@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=michal.simek@amd.com \
    --cc=radhey.shyam.pandey@amd.com \
    --cc=robh@kernel.org \
    --cc=shubhrajyoti.datta@amd.com \
    --cc=shubhrajyoti.datta@gmail.com \
    --cc=tony.luck@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox