[Linux-ia64] Re: [RFC] NUMA functions for accessing replicated areas

[Linux-ia64] Re: [RFC] NUMA functions for accessing replicated areas

[Andrea Arcangeli]

Hello Keith,

On Sat, Jan 19, 2002 at 11:25:17AM +1100, Keith Owens wrote:
> The linux kernel debugger (kdb) has hit a problem with setting software
> breakpoints on NUMA boxes with replicated kernel text.  Instead of
> changing one kernel virtual address and affecting the only copy of the
> kernel, kdb has to loop over each replicated area and change each copy
> of the kernel individually, using some mapping that gives access to the
> data on other nodes.
> 
> I don't want code in kdb that knows about every NUMA platform, that is
> a maintenance nightmare.  I want the NUMA platforms to provide the

Obviously a good idea.

> following interfaces.  Any objections or changes?  I will do the the
> SGI version, other manufacturers will have to provide the equivalent
> functions if they want to use kdb.
> 
> BTW, is there a single NUMA mailing list?  cc: me please, I am not on
> all lists.
> 
> CONFIG_NUMA_REPLICATE
> 
>   Set to y if the platform supports the replication functions.
> 
> int numa_replicated(unsigned long address, size)
> 
>   Return 0 if the area (address to address+size-1) is not replicated.
>   Return 1 if it is replicated.
>   Return -EINVAL if the area is a mixture of replicated and
>   non-replicated data.
> 
> int numa_replicate_loop(unsigned long address, (int *func)(void *numadata, void *funcdata), void *funcdata)
> 
>   Call the function for each replication of the address.  numadata
>   identifies this instance of the replicated data, it is an opaque
>   cookie to the function.  funcdata is supplied by the caller of
>   numa_replicate_loop(), it is an opaque cookie to the numa code.
> 
>   Return 0 if the function returns 0 for all calls, otherwise 1.  If
>   any call to func() returns non-zero, numa_replicate_loop() continues
>   with the other replicated areas.  If func() wants to ignore
>   subsequent areas it can set a flag in funcdata.
> 
> int numa_replicate_getarea_size(void *to, unsigned long from_xxx, size_t size, void *numadata)
> 
>   Read size bytes from from_xxx (which must be a replicated address) to
>   the user's data area.
> 
>   Return 0 on success, -EFAULT for invalid from_xxx address or size.
> 
> int numa_replicate_putarea_size(unsigned long to_xxx, void *from, size_t size, void *numadata)
> 
>   Write size bytes from the user's data area to to_xxx (which must be a
>   replicated address).
> 
>   Return 0 on success, -EFAULT for invalid to_xxx address or size.
> 
> Is that satisfactory?  Once those routines exist, kdb can use them to
> set software breakpoints.

The opaque void * spread across all the API looks rather scary.
Furthmore I'm not convinced we need to pass any cookie to the api, all
the calls knows the numadata info by the static status after boot.

So I think something like this should be enough:

	int numa_replicated(unsigned long address, int size);
	int numa_getarea(void *to, unsigned long from, int size);
	int numa_putarea(unsigned long to, void *from, int size);

Futhmore it may be even better to drop numa_replicated completly and to
default using numa_getarea/numa_putarea always, this should make kdb
even cleaner. Implementation of getarea/putarea for non numa case will
be a simple memcpy.

Andrea

[Linux-ia64] Re: [RFC] NUMA functions for accessing replicated areas

[Keith Owens]

On Sat, 19 Jan 2002 18:27:48 +0100, 
Andrea Arcangeli <andrea@suse.de> wrote:
>On Sat, Jan 19, 2002 at 11:25:17AM +1100, Keith Owens wrote:
>The opaque void * spread across all the API looks rather scary.
>Furthmore I'm not convinced we need to pass any cookie to the api, all
>the calls knows the numadata info by the static status after boot.
>
>So I think something like this should be enough:
>
>	int numa_replicated(unsigned long address, int size);
>	int numa_getarea(void *to, unsigned long from, int size);
>	int numa_putarea(unsigned long to, void *from, int size);

That assumes that all replicated data is identical.  It starts off that
way but what prevents one set of replicated data being changed and not
the others?  I don't mind that assumption, it makes life easier for kdb,
but I did not want to constrain NUMA implementations.  However if
everyone agrees that each instance of replcated data should always be
identical then numa_getarea() can read from any instance and
numa_putarea() writes to all instances, no need for numa_replicate_loop.

>Futhmore it may be even better to drop numa_replicated completly and to
>default using numa_getarea/numa_putarea always, this should make kdb
>even cleaner. Implementation of getarea/putarea for non numa case will
>be a simple memcpy.

If numa_replicated() returns true then it is safe to use memcpy, if
numa_replicated() is omitted then the get/put functions must validate
the from address on get and the to address on put.  kdb v2.1 relies on
the MMU via __copy_to_user() to detect invalid user supplied addresses.

[Linux-ia64] Re: [RFC] NUMA functions for accessing replicated areas

[Keith Owens]

On Sat, 19 Jan 2002 18:02:20 -0600 (CST), 
Jack Steiner <steiner@sgi.com> wrote:
>There is one thing that occurred to me. Currently, on IA64, the replicated
>data is read-execute. The NUMA code known how to write it but I dont
>know if you want to teach kdb how to do it.

That is exactly what I don't want to do.  I want numa_putarea_size to
take a virtual target address and copy the data to all instances of
that target address.  kdb should not know about translating one to
many, the numa functions hide that complexity from anything that needs
to update replicated data.