From: "Pandey, Radhey Shyam" <radheys@amd.com>
To: Shubhrajyoti Datta <shubhrajyoti.datta@amd.com>,
linux-edac@vger.kernel.org
Cc: git@amd.com, shubhrajyoti.datta@gmail.com, michal.simek@amd.com,
Radhey Shyam Pandey <radhey.shyam.pandey@amd.com>,
Krzysztof Kozlowski <krzk@kernel.org>,
Rob Herring <robh@kernel.org>, Conor Dooley <conor+dt@kernel.org>,
Borislav Petkov <bp@alien8.de>, Tony Luck <tony.luck@intel.com>,
linux-kernel@vger.kernel.org, devicetree@vger.kernel.org
Subject: Re: [PATCH v2 06/12] EDAC/versalnet: Wait for RPMsg endpoint initialization
Date: Sat, 21 Nov 2026 20:58:37 +0530 [thread overview]
Message-ID: <eb3ea65f-7602-48b5-b983-b09c091a816e@amd.com> (raw)
In-Reply-To: <20261118142213.100581-7-shubhrajyoti.datta@amd.com>
On 11/18/2026 7:50 PM, Shubhrajyoti Datta wrote:
> From: Hayden Rinn <haydenrinn@microsoft.com>
>
> The rpmsg endpoint can be used before rpmsg_probe() completes, and
> can race with asynchronous rpmsg_remove() teardown (e.g. R5 crash
> or shutdown), both leading to use of an uninitialized or freed
> endpoint.
>
> Add a completion so mc_probe() waits for the endpoint, and a
> "ready" flag so rpmsg_cb() drops events until mci/adec are
> set up. Add ept_lock to serialize priv->ept and priv->mcdi->ept
> across mcdi_request(), rpmsg_probe(), and rpmsg_remove(), clearing
> them on removal and failing mcdi_request() if the channel is down.
>
> Fixes: d5fe2fec6c40 ("EDAC: Add a driver for the AMD Versal NET DDR controller")
> Signed-off-by: Hayden Rinn <haydenrinn@microsoft.com>
Why Hayden is missed out in to/cc list?
> Co-developed-by: Shubhrajyoti Datta <shubhrajyoti.datta@amd.com>
> Signed-off-by: Shubhrajyoti Datta <shubhrajyoti.datta@amd.com>
> ---
>
> Changes in v2:
> Patch addition for the rpmsg endpoint race
>
> drivers/edac/versalnet_edac.c | 123 ++++++++++++++++++++++++++++++----
> 1 file changed, 111 insertions(+), 12 deletions(-)
>
> diff --git a/drivers/edac/versalnet_edac.c b/drivers/edac/versalnet_edac.c
> index ea90d3ce3e70..90918293f76d 100644
> --- a/drivers/edac/versalnet_edac.c
> +++ b/drivers/edac/versalnet_edac.c
> @@ -5,8 +5,10 @@
> */
>
> #include <linux/cdx/edac_cdx_pcol.h>
> +#include <linux/completion.h>
> #include <linux/edac.h>
> #include <linux/module.h>
> +#include <linux/mutex.h>
> #include <linux/of_device.h>
> #include <linux/ras.h>
> #include <linux/remoteproc.h>
> @@ -146,7 +148,22 @@ struct ecc_status {
> * @adec: Address decode registers.
> * @mci: Memory controller interface.
> * @ept: rpmsg endpoint.
> + * @ept_lock: Protects @ept and @mcdi->ept against concurrent teardown
> + * by rpmsg_remove() (e.g. remote processor crash/shutdown)
> + * while mcdi_request() is sending on them.
> + * @rpmsg_ready: Completion signaled once rpmsg_probe() has run to
> + * completion success. @ept is only valid to check
> + * once this is signaled.
> * @mcdi: The mcdi handle.
> + * @dev: Platform device backing this driver instance. Stays
> + * valid for as long as mcdi_request() can be called,
> + * unlike @mcdi->rpdev, which is torn down and cleared
> + * by rpmsg_remove(). Used for logging when there is no
> + * live rpmsg channel to hang the message off of.
To big description. Lets trim all this AI generated text.
> + * @ready: Set once edac priv data is initialised. Published with a
> + * release store and read with an acquire load so that rpmsg_cb
> + * never calls handle_error() against a @mci entry that hasn't
> + * been allocated yet.
Same here.
> */
> struct mc_priv {
> char message[256];
> @@ -159,7 +176,11 @@ struct mc_priv {
> u32 adec[ADEC_MAX];
> struct mem_ctl_info *mci[NUM_CONTROLLERS];
> struct rpmsg_endpoint *ept;
> + struct mutex ept_lock; /* Protects endpoint */
> + struct completion rpmsg_ready;
> struct cdx_mcdi *mcdi;
> + struct device *dev;
> + bool ready;
> };
>
> /*
> @@ -508,6 +529,12 @@ static void mc_init(struct mem_ctl_info *mci, struct device *dev,
>
> #define to_mci(k) container_of(k, struct mem_ctl_info, dev)
>
> +static struct rpmsg_device_id amd_rpmsg_id_table[] = {
> + { .name = EDAC_RPMSG_CHAN_NAME },
> + { },
> +};
> +MODULE_DEVICE_TABLE(rpmsg, amd_rpmsg_id_table);
> +
> static unsigned int mcdi_rpc_timeout(struct cdx_mcdi *cdx, unsigned int cmd)
> {
> return MCDI_RPC_TIMEOUT;
> @@ -517,6 +544,7 @@ static void mcdi_request(struct cdx_mcdi *cdx,
> const struct cdx_dword *hdr, size_t hdr_len,
> const struct cdx_dword *sdu, size_t sdu_len)
> {
> + struct mc_priv *mc_priv = (struct mc_priv *)amd_rpmsg_id_table[0].driver_data;
> void *send_buf;
> int ret;
>
> @@ -527,6 +555,24 @@ static void mcdi_request(struct cdx_mcdi *cdx,
> memcpy(send_buf, hdr, hdr_len);
> memcpy(send_buf + hdr_len, sdu, sdu_len);
>
> + guard(mutex)(&mc_priv->ept_lock);
> +
> + /*
> + * The rpmsg channel can be torn down asynchronously (e.g. the R5
> + * remote processor crashing or being shut down) by rpmsg_remove(),
> + * which clears both @cdx->ept and @mc_priv->ept under this same
> + * lock. Bail out instead of sending on a destroyed endpoint.
> + *
> + * @cdx->rpdev is cleared by rpmsg_remove() together with @cdx->ept,
> + * so it cannot be used for logging here; use the stable platform
> + * device instead.
> + */
> + if (!cdx->ept) {
> + dev_err(mc_priv->dev, "Dropping rpmsg send, channel is down\n");
> + kfree(send_buf);
Better to do in start and avoid this kfree.
> + return;
> + }
> +
> ret = rpmsg_send(cdx->ept, send_buf, hdr_len + sdu_len);
> if (ret)
> dev_err(&cdx->rpdev->dev, "Failed to send rpmsg data: %d\n", ret);
> @@ -601,6 +647,18 @@ static int rpmsg_cb(struct rpmsg_device *rpdev, void *data,
> cdx_mcdi_process_cmd(mc_priv->mcdi, (struct cdx_dword *)data, len);
> return 0;
> }
> + /*
> + * The rpmsg channel (and thus this callback) can become live
> + * before init_versalnet() has allocated priv->mci[] and
> + * populated priv->adec[]. Drop reports that race ahead
> + * of it instead of calling handle_error() against incomplete
> + * state.
> + */
> + if (!smp_load_acquire(&mc_priv->ready)) {
> + dev_warn_ratelimited(&rpdev->dev,
> + "Dropping event, driver not ready yet\n");
> + return 0;
> + }
>
> sec_sev = result[ERROR_LEVEL];
> error_id = result[ERROR_ID];
> @@ -721,12 +779,6 @@ static int rpmsg_cb(struct rpmsg_device *rpdev, void *data,
> return 0;
> }
>
> -static struct rpmsg_device_id amd_rpmsg_id_table[] = {
> - { .name = EDAC_RPMSG_CHAN_NAME },
> - { },
> -};
> -MODULE_DEVICE_TABLE(rpmsg, amd_rpmsg_id_table);
> -
> static int rpmsg_probe(struct rpmsg_device *rpdev)
> {
> struct rpmsg_channel_info chinfo = {
> @@ -734,25 +786,56 @@ static int rpmsg_probe(struct rpmsg_device *rpdev)
> .dst = rpdev->dst,
> .name = EDAC_RPMSG_CHAN_NAME,
> };
> + struct rpmsg_endpoint *ept;
> struct mc_priv *pg;
>
> pg = (struct mc_priv *)amd_rpmsg_id_table[0].driver_data;
>
> - pg->ept = rpmsg_create_ept(rpdev, rpmsg_cb, NULL, chinfo);
> - if (!pg->ept)
> - return dev_err_probe(&rpdev->dev, -ENXIO, "Failed to create ept for channel %s\n",
> - chinfo.name);
> -
> dev_set_drvdata(&rpdev->dev, pg);
>
> + ept = rpmsg_create_ept(rpdev, rpmsg_cb, NULL, chinfo);
> + if (!ept) {
> + return dev_err_probe(&rpdev->dev, -ENXIO,
> + "Failed to create ept for channel %s\n",
> + chinfo.name);
> + }
> +
> + /*
> + * Link the new endpoint in under the lock. This also re-syncs
> + * mcdi->ept on reconnection after the remote processor recovers
> + * from a crash/shutdown, which previously only happened once,
> + * during the initial mc_probe().
> + */
> + scoped_guard(mutex, &pg->ept_lock) {
> + pg->ept = ept;
> + pg->mcdi->ept = ept;
> + pg->mcdi->rpdev = rpdev;
> + }
[1] Critical: rpmsg_probe() dereferences priv->mcdi before mc_probe()
assigns it.
priv->mcdi is assigned inside setup_mcdi(), which mc_probe() calls
after register_rpmsg_driver(). If the channel binds in that window -
only a kzalloc_obj() and cdx_mcdi_init() wide, but the NS callback
runs in an unrelated context - pg->mcdi is NULL here.
Pre-patch rpmsg_probe() only touched pg->ept, which lives in the
devm-allocated priv, so this is newly introduced.
There is a second, wider instance: if setup_mcdi() fails in
cdx_mcdi_init() it returns with mc_priv->mcdi still NULL, and
"goto err_unreg" runs unregister_rpmsg_driver() -> rpmsg_remove(),
which does mc_priv->mcdi->ept = NULL on NULL.
> +
> + complete_all(&pg->rpmsg_ready);
> +
> return 0;
> }
>
> static void rpmsg_remove(struct rpmsg_device *rpdev)
> {
> struct mc_priv *mc_priv = dev_get_drvdata(&rpdev->dev);
> + struct rpmsg_endpoint *ept;
> +
> + /*
> + * Clear the shared endpoint pointers before destroying the
> + * endpoint, so that mcdi_request() (serialised on the same lock)
> + * cannot observe or send on a stale/freed endpoint if the remote
> + * processor goes down concurrently.
> + */
> + scoped_guard(mutex, &mc_priv->ept_lock) {
> + ept = mc_priv->ept;
> + mc_priv->ept = NULL;
> + mc_priv->mcdi->ept = NULL;
> + mc_priv->mcdi->rpdev = NULL;
[2] Critical: use-after-free on the probe error path.
> @@ -745,25 +786,56 @@ static void rpmsg_remove(struct rpmsg_device
*rpdev)
> + scoped_guard(mutex, &mc_priv->ept_lock) {
> + ept = mc_priv->ept;
> + mc_priv->ept = NULL;
> + mc_priv->mcdi->ept = NULL;
> + mc_priv->mcdi->rpdev = NULL;
> + }
Combined with the unwind in mc_probe(), which is unchanged by this
patch but newly matters because rpmsg_remove() now writes through
priv->mcdi:
err_init:
cdx_mcdi_finish(priv->mcdi);
kfree(priv->mcdi);
err_unreg:
unregister_rpmsg_driver(&amd_rpmsg_driver);
err_init frees priv->mcdi and then falls through to
unregister_rpmsg_driver(), which calls rpmsg_remove() on the still
bound channel, which writes to the freed object. Reachable whenever
init_versalnet() fails, since the channel is bound by then.
mc_remove() has the right order (unregister, then free); only the
error path is inverted.
> + }
>
> - rpmsg_destroy_ept(mc_priv->ept);
> + rpmsg_destroy_ept(ept);
> dev_set_drvdata(&rpdev->dev, NULL);
> }
>
> @@ -920,6 +1003,9 @@ static int mc_probe(struct platform_device *pdev)
> goto err_alloc;
> }
>
> + priv->dev = &pdev->dev;
> + mutex_init(&priv->ept_lock);
> + init_completion(&priv->rpmsg_ready);
> amd_rpmsg_id_table[0].driver_data = (kernel_ulong_t)priv;
>
> rc = register_rpmsg_driver(&amd_rpmsg_driver);
> @@ -932,12 +1018,25 @@ static int mc_probe(struct platform_device *pdev)
> if (rc)
> goto err_unreg;
>
[3] Critical: the completion is waited on after the endpoint's only
pre-init user, so the patch does not fix the bug it describes.
The wait lands after setup_mcdi(), but setup_mcdi() is itself the
first and only user of the endpoint during probe:
mcdi_request() takes the new !cdx->ept exit and returns void,
so the CDX core never learns the send failed and waits out
cdx_mcdi_rpc_timeout(), 10 * HZ. get_ddr_config() only copies on
success and setup_mcdi() returns 0 regardless, so adec[] stays zero and
probe continues: init_one_mc() reads FIELD_GET(MC5_BUS_WIDTH_MASK, 0)
== XDDR5_BUS_WIDTH_64 and registers MC0 with fabricated geometry. (The
announcement normally lands during that stall, so in practice only MC0
is affected.)
Pre-patch this at least produced a WARN_ON(!ept) from rpmsg_send(). The
guard is correct, but without moving the wait above the loop it only
silences the symptom.
> + if (!wait_for_completion_timeout(&priv->rpmsg_ready, MCDI_RPC_TIMEOUT)) {
> + dev_err(&pdev->dev, "Timed out waiting for RPMsg endpoint\n");
> + rc = -ENODEV;
> + goto err_init;
> + }
> +
> priv->mcdi->r5_rproc = rp;
>
> rc = init_versalnet(priv, pdev);
> if (rc)
> goto err_init;
>
> + /*
> + * edac is initialised. Publish this with a release store so rpmsg_cb
> + * running concurrently on the rpmsg worker thread, only calls
> + * handle_error() once it observes @ready via a matching acquire load.
> + */
> + smp_store_release(&priv->ready, true);
> +
> return 0;
>
> err_init:
next prev parent reply other threads:[~2026-11-21 15:29 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 14:20 [PATCH v2 00/12] EDAC/versalnet: Fix error handling, teardown, and robustness Shubhrajyoti Datta
2026-11-18 14:20 ` [PATCH v2 01/12] EDAC/versalnet: Fix device_register() error handling in init_one_mc() Shubhrajyoti Datta
2026-11-20 13:38 ` Pandey, Radhey Shyam
2026-11-18 14:20 ` [PATCH v2 02/12] EDAC/versalnet: Use designated initializer for rpmsg_channel_info Shubhrajyoti Datta
2026-11-20 13:39 ` Pandey, Radhey Shyam
2026-11-18 14:20 ` [PATCH v2 03/12] EDAC/versalnet: Fix use-after-free in remove_one_mc() Shubhrajyoti Datta
2026-11-18 14:20 ` [PATCH v2 04/12] EDAC/versalnet: Add bounds validation in rpmsg_cb() Shubhrajyoti Datta
2026-11-18 14:33 ` sashiko-bot
2026-11-20 12:18 ` Datta, Shubhrajyoti
2026-11-20 15:43 ` Pandey, Radhey Shyam
2026-11-18 14:20 ` [PATCH v2 05/12] EDAC/versalnet: Make dwidth per-controller Shubhrajyoti Datta
2026-11-20 16:54 ` Pandey, Radhey Shyam
2026-11-18 14:20 ` [PATCH v2 06/12] EDAC/versalnet: Wait for RPMsg endpoint initialization Shubhrajyoti Datta
2026-11-21 15:28 ` Pandey, Radhey Shyam [this message]
2026-11-18 14:20 ` [PATCH v2 07/12] EDAC/versalnet: Initialize MCDI before RPMsg registration Shubhrajyoti Datta
2026-11-21 15:33 ` Pandey, Radhey Shyam
2026-11-18 14:20 ` [PATCH v2 08/12] EDAC/versalnet: Guard against NULL drvdata in rpmsg_cb() Shubhrajyoti Datta
2026-11-18 14:20 ` [PATCH v2 09/12] EDAC/versalnet: Move platform_set_drvdata() to mc_probe() Shubhrajyoti Datta
2026-11-18 14:20 ` [PATCH v2 10/12] dt-bindings: memory: xlnx,versal-net-ddrmc5: Add amd,num-ctrl property Shubhrajyoti Datta
2026-11-18 14:20 ` [PATCH v2 11/12] EDAC/versalnet: use device tree property for number of controllers Shubhrajyoti Datta
2026-11-18 14:20 ` [PATCH v2 12/12] EDAC/versalnet: Make get_ddr_config() return errors Shubhrajyoti Datta
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=eb3ea65f-7602-48b5-b983-b09c091a816e@amd.com \
--to=radheys@amd.com \
--cc=bp@alien8.de \
--cc=conor+dt@kernel.org \
--cc=devicetree@vger.kernel.org \
--cc=git@amd.com \
--cc=krzk@kernel.org \
--cc=linux-edac@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=michal.simek@amd.com \
--cc=radhey.shyam.pandey@amd.com \
--cc=robh@kernel.org \
--cc=shubhrajyoti.datta@amd.com \
--cc=shubhrajyoti.datta@gmail.com \
--cc=tony.luck@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox