[PATCH 0/2] ia64: Migrate data off physical pages with correctable errors

[PATCH 0/2] ia64: Migrate data off physical pages with correctable errors

[Russ Anderson]

	Migrate data off physical pages with corrected memory errors

Purpose:

	Physical memory with corrected errors may decay over time into
	uncorrectable errors.  The purpose of this patch is to move the
	data off pages with correctable memory errors before the memory
	goes bad.

The patches:

  [1/2] page.discard: Avoid putting a bad page back on the LRU.

	page.discard are the arch independent changes.  It adds a new
	page flag (PG_memerror) to mark the page as bad and prevent it
	from being put back on the LRU.  PG_memerror is bit 32 and only
	defined on 64 bit architectures.  It adds "BadPages:" output to
	/proc/meminfo on 64 bit architectures.  

  [2/2] cpe.migrate: Call migration code on correctable errors

	cpe.migrate are the IA64 specific changes.  It connects the CPE
	handler to the page migration code.  It is implemented as a kernel
	loadable module, similar to the mca recovery code (mca_recovery.ko),
	so that it can be removed to turn off the feature.  It exports
	three symbols (migrate_prep, isolate_lru_page, and migrate_pages).

Comments:

	Since page flags are a precious commodity on 32 bit architectures,
	the choice was made to implement PG_memerror only on 64 bit 
	architectures, in the upper 32 bits.  If there is interest in
	this feature on 32 bit, it only requires defining PG_memerror
	in one of the lower 32 page flage bits and removing the BITS_PER_LONG
	checks.

	There is always an issue of how agressive the code should be on
	migrating pages.  Should it migrate on the first correctable error,
	or wait for some threshold?  Reasonable people may disagree on the
	threshold and the "right" answer may be hardware specific.  The 
	decision making is confined to the cpe_migrate.c code.  It is
	currently set to migrate on the first correctable error.

	Only pages that can be isolated on the LRU are migrated.  Other
	pages, such as compound pages, are not migrated.  That functionality
	could be a future enhancement.

Flow of the code description (while testing on IA64):

	1) A user level application test program allocates memory and
	   passes the virtual address of the memory to the error injection
	   driver.

	2) The error injection driver converts the virtual address to
	   physical, functions the Altix hardware to modify the ECC for the
	   physical page, creating a correctable error, and returns to the
	   user application.

	3) The user application reads the memory.

	4) The Altix hardware detects the correctable error and interrupts
	   prom.  SAL builds a CPU error record, then sends a CPE 
	   interrupt to linux.

	5) The linux CPE handler calls the cpe_migrate module (if installed).

	6) cpe_migrate parses the physical address from the CPE record and
	   adds the address to the migrate list (if not already on the list)
	   and schedules the worker thread (cpe_enable_work).

	7) cpe_enable_work calls ia64_mca_cpe_move_page.

	8) ia64_mca_cpe_move_page validates the physical address, converts
	   to page, sets PG_memerror flag and calls the migration code
	   (migrate_prep(), isolate_lru_page(), and migrate_pages().  If the
	   page migrates successfully, the bad page is added to badpagelist.

	9) Because PG_memerror is set, the bad page is not added back on the LRU
	   due to checks in lru_cache_add() and lru_cache_add_active().

-- 
Russ Anderson, OS RAS/Partitioning Project Lead  
SGI - Silicon Graphics Inc          rja@sgi.com

Re: [PATCH 0/2] ia64: Migrate data off physical pages with correctable errors

[Matthew Wilcox]

On Mon, Apr 28, 2008 at 02:22:52PM -0500, Russ Anderson wrote:
> 	There is always an issue of how agressive the code should be on
> 	migrating pages.  Should it migrate on the first correctable error,
> 	or wait for some threshold?  Reasonable people may disagree on the
> 	threshold and the "right" answer may be hardware specific.  The 
> 	decision making is confined to the cpe_migrate.c code.  It is
> 	currently set to migrate on the first correctable error.

I think the kernel code should do the migration ASAP.  But I think we
should have a list of 'bad' pages.  We could then have a badram driver
that userspace can talk to to find out which pages are bad, map those
pages into a badram process, do various tests on them, and return the
pages to the pool if they're determined to be 'good'.

I could also see badramd having a list of pages found to be bad
in previous boots and asking the badram driver to take them out of
circulation early in boot before they've been allocated.

-- 
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [PATCH 0/2] ia64: Migrate data off physical pages with correctable errors

[Russ Anderson]

On Mon, Apr 28, 2008 at 01:33:23PM -0600, Matthew Wilcox wrote:
> On Mon, Apr 28, 2008 at 02:22:52PM -0500, Russ Anderson wrote:
> > 	There is always an issue of how agressive the code should be on
> > 	migrating pages.  Should it migrate on the first correctable error,
> > 	or wait for some threshold?  Reasonable people may disagree on the
> > 	threshold and the "right" answer may be hardware specific.  The 
> > 	decision making is confined to the cpe_migrate.c code.  It is
> > 	currently set to migrate on the first correctable error.
> 
> I think the kernel code should do the migration ASAP.  But I think we
> should have a list of 'bad' pages.  We could then have a badram driver
> that userspace can talk to to find out which pages are bad, map those
> pages into a badram process, do various tests on them, and return the
> pages to the pool if they're determined to be 'good'.

Sure.  The bad page list is badpagelist (defined in mca.c).
 
> I could also see badramd having a list of pages found to be bad
> in previous boots and asking the badram driver to take them out of
> circulation early in boot before they've been allocated.

That would be one alternative.  That type functionality would be
useful.  FWIW, some of my testing was on a system with a DIMM with
solid single bits.  It is a row/column problem so several meg of
addresses are effected.  Each boot it would migrate several meg worth
of pages without any problem.  It works surprisingly good.

-- 
Russ Anderson, OS RAS/Partitioning Project Lead  
SGI - Silicon Graphics Inc          rja@sgi.com