Re: [RFC PATCH] cxl: avoid duplicating report from MCE & device

[Dan Williams <dan.j.williams@intel.com>]

Jonathan Cameron wrote:
> On Wed, 19 Jun 2024 00:53:10 +0800
> Shiyang Ruan <ruansy.fnst@fujitsu.com> wrote:
> 
> > Background:
> > Since CXL device is a memory device, while CPU consumes a poison page of 
> > CXL device, it always triggers a MCE by interrupt (INT18), no matter 
> > which-First path is configured.  This is the first report.  Then 
> > currently, in FW-First path, the poison event is transferred according 
> > to the following process: CXL device -> firmware -> OS:ACPI->APEI->GHES 
> >  -> CPER -> trace report.  This is the second one.  These two reports  
> > are indicating the same poisoning page, which is the so-called "duplicate
> > report"[1].  And the memory_failure() handling I'm trying to add in
> > OS-First path could also be another duplicate report.
> > 
> > Hope the flow below could make it easier to understand:
> > CPU accesses bad memory on CXL device, then
> >  -> MCE (INT18), *always* report (1)
> >  -> * FW-First (implemented now)
> >       -> CXL device -> FW
> > 	      -> OS:ACPI->APEI->GHES->CPER -> trace report (2.a)  
> >     * OS-First (not implemented yet, I'm working on it)
> >       -> CXL device -> MSI
> > 	      -> OS:CXL driver -> memory_failure() (2.b)  
> > so, the (1) and (2.a/b) are duplicated.
> > 
> > (I didn't get response in my reply for [1] while I have to make patch to
> > solve this problem, so please correct me if my understanding is wrong.)
> > 
> > This patch adds a new notifier_block and MCE_PRIO_CXL, for CXL memdev
> > to check whether the current poison page has been reported (if yes,
> > stop the notifier chain, won't call the following memory_failure()
> > to report), into `x86_mce_decoder_chain`.  In this way, if the poison
> > page already handled(recorded and reported) in (1) or (2), the other one
> > won't duplicate the report.  The record could be clear when
> > cxl_clear_poison() is called.
> > 
> > [1] https://lore.kernel.org/linux-cxl/664d948fb86f0_e8be294f8@dwillia2-mobl3.amr.corp.intel.com.notmuch/
> > 
> > Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> 
> So poison can be cleared in a number of ways and a CXL poison clear command
> is unfortunately only one of them.  Some architectures have instructions
> that guarantee to write a whole cacheline and can clear things as well.
> I believe x86 does for starters.

Yes, movdir64b.

> +CC linux-edac and related maintainers / reviewers.
>     linux-mm and hwpoison maintainer.
> 
> So I think this needs a more general solution that encompasses 
> more general cleanup of poison.

I think unless the device has "List Poison" coverage for volatile ranges
that the kernel should not worry about tracking this itself.

Perhaps what is needed is that after successful memory_failure()
handling when the page is known to be offline the device backing the
memory can be notified that it is safe to repair the page and but it
back into service, but I expect that would be comparison of the device's
own poison tracking relative to the notification of successful page
offline.

> 
> Trivial comments inline.
> 
> Jonathan
> 
> 
> > ---
> >  arch/x86/include/asm/mce.h |   1 +
> >  drivers/cxl/core/mbox.c    | 130 +++++++++++++++++++++++++++++++++++++
> >  drivers/cxl/core/memdev.c  |   6 +-
> >  drivers/cxl/cxlmem.h       |   3 +
> >  4 files changed, 139 insertions(+), 1 deletion(-)
> > 
> > diff --git a/arch/x86/include/asm/mce.h b/arch/x86/include/asm/mce.h
> > index dfd2e9699bd7..d8109c48e7d9 100644
> > --- a/arch/x86/include/asm/mce.h
> > +++ b/arch/x86/include/asm/mce.h
> > @@ -182,6 +182,7 @@ enum mce_notifier_prios {
> >  	MCE_PRIO_NFIT,
> >  	MCE_PRIO_EXTLOG,
> >  	MCE_PRIO_UC,
> > +	MCE_PRIO_CXL,
> >  	MCE_PRIO_EARLY,
> >  	MCE_PRIO_CEC,
> >  	MCE_PRIO_HIGHEST = MCE_PRIO_CEC
> > diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> > index 2626f3fff201..0eb3c5401e81 100644
> > --- a/drivers/cxl/core/mbox.c
> > +++ b/drivers/cxl/core/mbox.c
> > @@ -4,6 +4,8 @@
> >  #include <linux/debugfs.h>
> >  #include <linux/ktime.h>
> >  #include <linux/mutex.h>
> > +#include <linux/notifier.h>
> > +#include <asm/mce.h>
> >  #include <asm/unaligned.h>
> >  #include <cxlpci.h>
> >  #include <cxlmem.h>
> > @@ -880,6 +882,9 @@ void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
> >  		if (cxlr)
> >  			hpa = cxl_trace_hpa(cxlr, cxlmd, dpa);
> >  
> > +		if (hpa != ULLONG_MAX && cxl_mce_recorded(hpa))
> > +			return;
> > +
> >  		if (event_type == CXL_CPER_EVENT_GEN_MEDIA)
> >  			trace_cxl_general_media(cxlmd, type, cxlr, hpa,
> >  						&evt->gen_media);
> > @@ -1408,6 +1413,127 @@ int cxl_poison_state_init(struct cxl_memdev_state *mds)
> >  }
> >  EXPORT_SYMBOL_NS_GPL(cxl_poison_state_init, CXL);
> >  
> > +struct cxl_mce_record {
> > +	struct list_head node;
> > +	u64 hpa;
> > +};
> > +LIST_HEAD(cxl_mce_records);
> > +DEFINE_MUTEX(cxl_mce_mutex);
> > +
> > +bool cxl_mce_recorded(u64 hpa)
> > +{
> > +	struct cxl_mce_record *cur, *next, *rec;
> > +	int rc;
> > +
> > +	rc = mutex_lock_interruptible(&cxl_mce_mutex);
> 
> guard(mutex)(&cxl_mce_muted);

Agree, _interruptible is really only suitable for user ABI facing locks,
not kernel internal helper functions, but this comment is moot if this
tracking switches to xarray.

> 
> > +	if (rc)
> > +		return false;
> > +
> > +	list_for_each_entry_safe(cur, next, &cxl_mce_records, node) {
> > +		if (cur->hpa == hpa) {
> > +			mutex_unlock(&cxl_mce_mutex);
> > +			return true;
> > +		}
> > +	}
> > +
> > +	rec = kmalloc(sizeof(struct cxl_mce_record), GFP_KERNEL);
> > +	rec->hpa = hpa;
> > +	list_add(&cxl_mce_records, &rec->node);
> > +
> > +	mutex_unlock(&cxl_mce_mutex);
> > +
> > +	return false;
> > +}
> > +
> > +void cxl_mce_clear(u64 hpa)
> > +{
> > +	struct cxl_mce_record *cur, *next;
> > +	int rc;
> > +
> > +	rc = mutex_lock_interruptible(&cxl_mce_mutex);
> 
> Maybe cond_guard().

cond_guard() was rejected, you meant scoped_cond_guard()? But, then I
think _interruptible is not appropriate here.