per-thread global variables

per-thread global variables

[J.A. Magallon]

Hi all.

I am looking for a method to have a global variable that has a unique
personality fot each thread. I don't want to use pthreads, but I would
like to emulate [get,set]specific. I just want to use clone() directly.

I have read something about a __thread variable modifier, or something
about an __attribute(( )) in gcc. Other solutions imply a search based
on pid, but I would like to find some more direct method.
Something like using the PRDA in IRIX.

Anybody has any pointer/info on this ?

TIA

-- 
J.A. Magallon             \   Software is like sex: It's better when it's free
mailto:jamagallon@able.es  \                    -- Linus Torvalds, FSF T-shirt
Linux werewolf 2.4.19-rc1-jam3, Mandrake Linux 8.3 (Cooker) for i586
gcc (GCC) 3.1.1 (Mandrake Linux 8.3 3.1.1-0.7mdk)

Re: per-thread global variables

[Alan Cox]

> about an __attribute(( )) in gcc. Other solutions imply a search based
> on pid, but I would like to find some more direct method.
> Something like using the PRDA in IRIX.

If your stacks are the same size you can do the kernel trick with stack
maths to hide thread globals. 

Re: per-thread global variables

[J.A. Magallon]

On 2002.07.12 Alan Cox wrote:
>> about an __attribute(( )) in gcc. Other solutions imply a search based
>> on pid, but I would like to find some more direct method.
>> Something like using the PRDA in IRIX.
>
>If your stacks are the same size you can do the kernel trick with stack
>maths to hide thread globals. 
>-
>To unsubscribe from this list: send the line "unsubscribe linux-smp" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Uhm, something like reserving X bytes for stack, and telling clone()
I pass it a X-sizeof(prda) stack ?

How do I get the stack start from inside a thread ?

BTW, isn't there a barrier() syscall in Linux ? I could implement it 
with a semaphore with -N holes, but...man sem_init

       tially to value.  The pshared argument  indicates  whether
       the semaphore is local to the current process ( pshared is
       zero) or is to  be  shared  between  several  processes  (
       pshared is not zero). LinuxThreads currently does not sup­
       port  process-shared  semaphores,  thus  sem_init   always
       returns with error ENOSYS if pshared is not zero.


TIA

-- 
J.A. Magallon             \   Software is like sex: It's better when it's free
mailto:jamagallon@able.es  \                    -- Linus Torvalds, FSF T-shirt
Linux werewolf 2.4.19-rc1-jam3, Mandrake Linux 8.3 (Cooker) for i586
gcc (GCC) 3.1.1 (Mandrake Linux 8.3 3.1.1-0.7mdk)

Re: per-thread global variables

[Alan Cox]

> Uhm, something like reserving X bytes for stack, and telling clone()
> I pass it a X-sizeof(prda) stack ?

Use a 32K (say) 32K aligned stack
Load %esp into %eax, mask off the lower bits


> BTW, isn't there a barrier() syscall in Linux ? I could implement it 
> with a semaphore with -N holes, but...man sem_init

Too slow, way too slow. Read the x86 architecture manual and do it nicely 8)

Re: per-thread global variables

[Robert M. Hyatt]

I do this in Crafty.  The approach I chose is to create a structure
for each thread (I also use clone() since the glibc guys broke 
pthreads a while back) and then let each thread set its pointer to
its own private structure (really shared, but since only one thread
has a pointer to each structure, it is like a private but global
group of variables).

Doing it like this will guarantee it will work with any compiler,
which might be a plus...

On 
Fri, 12 Jul 2002, J.A. Magallon wrote:

> Hi all.
> 
> I am looking for a method to have a global variable that has a unique
> personality fot each thread. I don't want to use pthreads, but I would
> like to emulate [get,set]specific. I just want to use clone() directly.
> 
> I have read something about a __thread variable modifier, or something
> about an __attribute(( )) in gcc. Other solutions imply a search based
> on pid, but I would like to find some more direct method.
> Something like using the PRDA in IRIX.
> 
> Anybody has any pointer/info on this ?
> 
> TIA
> 
> 

-- 
Robert Hyatt                    Computer and Information Sciences
hyatt@cis.uab.edu               University of Alabama at Birmingham
(205) 934-2213                  115A Campbell Hall, UAB Station 
(205) 934-5473 FAX              Birmingham, AL 35294-1170

Re: per-thread global variables

[J.A. Magallon]

On 2002.07.12 Robert M. Hyatt wrote:
>
>
>I do this in Crafty.  The approach I chose is to create a structure
>for each thread (I also use clone() since the glibc guys broke 
>pthreads a while back) and then let each thread set its pointer to
>its own private structure (really shared, but since only one thread
>has a pointer to each structure, it is like a private but global
>group of variables).
>
>Doing it like this will guarantee it will work with any compiler,
>which might be a plus...
>

I think something is very dark for me...The hard way in pseudo-C

short id[PID_MAX];
#define setself(k) do { tid[getpid()]=k; } while(0)
#define self() (id[getpid()])

// master
  setself(-1);
// slaves
  for (i in 0..nslaves-1)
    clone(f(),CLONE_VM,i);

f(void *arg)
{
  setself((int)arg);
  print(self());
}

But how about the 32K*2 array ??? (hash table, ordered vector and binary
search...)

Quoting your answer:

>pthreads a while back) and then let each thread set its pointer to
                                                     ^^^
>its own private structure (really shared, but since only one thread

How do you define 'its' pointer ? That is like if I just defined

short id;

and tried to set 'its id' for each thread ?
Whichever method you use, at the end you always need one, the last, the
unique variable (one int, a pointer to a data area), declared as a global
variable in C that you need to be distinct for each thread. The only
thing (in my short undestanding) that can distinguish a thread from one
other is _pid_. Of course, I don't want to mess passing an info struct
to all functions in my code.

On IRIX, you have a

// Fixed address space always private for each process/thread
#define PRDA        ((struct prda *)0x00200000L)

struct prda*	thrprda;
int*	self;

f(int k)
{
  thrprda = (struct prda*)PRDA; // The magic global pointer
                                // different for each thread
  self = &(thrprda->user_area);
  *self = k;
}

If Linux supported this it will make many many things simpler.
A fixed zone of virtual address space that is never shared.

Or perhaps... Could I do in Linux:

#define PRDA        ((void*)0x00200000L) // Or get any free address space

int* self = PRDA; // will be shared over clone(), so we fix it before

for (i in 0..nslaves-1)
  clone(f(),CLONE_VM,i);

f(void *arg)
{
  // Get a chunk, mapped on a fixed address, and do not propagate the
  // map to parent or siblings
  mmap(self,1024,PROT_??,MAP_ANONYMOUS|MAP_FIXED|MAP_PRIVATE,0,0);
  *self = (int)arg;
}


??

TIA

-- 
J.A. Magallon             \   Software is like sex: It's better when it's free
mailto:jamagallon@able.es  \                    -- Linus Torvalds, FSF T-shirt
Linux werewolf 2.4.19-rc1-jam3, Mandrake Linux 8.3 (Cooker) for i586
gcc (GCC) 3.1.1 (Mandrake Linux 8.3 3.1.1-0.7mdk)

Re: per-thread global variables

[Alan Cox]

On Sat, 2002-07-13 at 01:12, J.A. Magallon wrote:
> On IRIX, you have a
> 
> // Fixed address space always private for each process/thread
> #define PRDA        ((struct prda *)0x00200000L)
> 
> struct prda*	thrprda;
> int*	self;
> 

Which goes to show they weren't planning for clone like performance. A
single unshared page means two seperate mm structs, two sets of page
tables and switches causing TLB flushes. All because

	movl %esp, %eax
	andl $ffff8000, %eax
	
and casting that is 'hard'

Alan

Re: per-thread global variables

[J.A. Magallon]

On 2002.07.13 Alan Cox wrote:
>On Sat, 2002-07-13 at 01:12, J.A. Magallon wrote:
>> On IRIX, you have a
>> 
>> // Fixed address space always private for each process/thread
>> #define PRDA        ((struct prda *)0x00200000L)
>> 
>> struct prda*	thrprda;
>> int*	self;
>> 
>
>Which goes to show they weren't planning for clone like performance. A
>single unshared page means two seperate mm structs, two sets of page
>tables and switches causing TLB flushes. All because
>
>	movl %esp, %eax
>	andl $ffff8000, %eax
>	
>and casting that is 'hard'
>

OK, I do not know a word on x86 assembler, but lets see if at least I understand
this.

stack = reserve 32K+sizeof(my_private_area) (say, 1Kb)
clone() lying about stack start: stack + 32K - 1
   (private on top and stack grows downwards)

   |---- stack ----||--- private -----|
   0                32K               33k
                   ^
                   stack passed to clone()

void* getpriv()
{
  // get my stack frame pointer, can be anywhere depending on my
  // 
  movl %esp, %eax
  |---- stack ----||--- private -----|
           ^
           %eax
  // Round it to 32k multiple, cause I know the thread stack is 32k aligned
  // So I have the thread stack end, ie, bottom
  andl $ffff8000, %eax
  |---- stack ----||--- private -----|
  ^
  %eax
  // Move after top of stack
  addl 32K,%eax
  |---- stack ----||--- private -----|
                   ^
                   %eax
  // and I have the begining of my private area, upwards
}

Could it even be correct ?

TIA

-- 
J.A. Magallon             \   Software is like sex: It's better when it's free
mailto:jamagallon@able.es  \                    -- Linus Torvalds, FSF T-shirt
Linux werewolf 2.4.19-rc1-jam3, Mandrake Linux 8.3 (Cooker) for i586
gcc (GCC) 3.1.1 (Mandrake Linux 8.3 3.1.1-0.7mdk)

Re: per-thread global variables

[J.A. Magallon]

On 2002.07.13 J.A. Magallon wrote:
>
>>
>>Which goes to show they weren't planning for clone like performance. A
>>single unshared page means two seperate mm structs, two sets of page
>>tables and switches causing TLB flushes. All because
>>
>>	movl %esp, %eax
>>	andl $ffff8000, %eax
>>	
>>and casting that is 'hard'
>>

It can be done even arch-independent (__builtin_frame_address is at least
since 2.96):

#include <sched.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <malloc.h>
#include <sys/user.h>

#define MAXT 4

// User definable
#define STACK_PAGES 8
#define PRIV_PAGES 1
//

#define STACK_SIZE (STACK_PAGES*PAGE_SIZE)
#define PRIV_SIZE (PRIV_PAGES*PAGE_SIZE)
#define TOTAL_PAGES (STACK_PAGES+PRIV_PAGES)
#define TOTAL_SIZE (TOTAL_PAGES*PAGE_SIZE)
#define TOTAL_MASK (~(TOTAL_SIZE-1))

void* getpriv();
#define self() (*(int*)getpriv())
#define setself(k) do { *(int*)getpriv() = k; } while(0)

int slave(void *arg);
void f();

int main(int argc,char** argv)
{
	void*	stack[MAXT];
	int		i;

	for (i=0; i<MAXT; i++)
	{
		stack[i] = memalign(TOTAL_SIZE,TOTAL_SIZE);
		clone(slave,stack[i]+STACK_SIZE-1,CLONE_VM|SIGCHLD,(void*)i);
	}
	for (i=0; i<MAXT; i++)
		wait(0);

	for (i=0; i<MAXT; i++)
		free(stack[i]);

	return 0;
}

void* getpriv()
{
	void* frame = __builtin_frame_address(0);
	void* priv = (void*)((unsigned long)frame&TOTAL_MASK);

	return priv+(ptrdiff_t)STACK_SIZE;
}

int slave(void *arg)
{
	setself((int)arg);
	printf("priv=%p\n",getpriv());
	printf("self=%d\n",self());

	f();

	return 0;
}

void f()
{
	printf("f self=%d\n",self());
}

Code for getpriv():

getpriv:
    pushl   %ebp
    subl    $8, %esp
    movl    %ebp, 4(%esp)
    movl    4(%esp), %eax
    andl    $-36864, %eax
    movl    %eax, (%esp)
    movl    (%esp), %eax
    addl    $32768, %eax
    addl    $8, %esp
    popl    %ebp
    ret

So it does not look too slow. I could do also sysconf(_SC_PAGESIZE), but
that adds a function call everywhere. Indeed it will be safer.

And you could fill private area before clone(), so children do not have to
worry about anything like setself(). Yup, it would be nicer...

Thanks for everything !!!

-- 
J.A. Magallon             \   Software is like sex: It's better when it's free
mailto:jamagallon@able.es  \                    -- Linus Torvalds, FSF T-shirt
Linux werewolf 2.4.19-rc1-jam3, Mandrake Linux 8.3 (Cooker) for i586
gcc (GCC) 3.1.1 (Mandrake Linux 8.3 3.1.1-0.7mdk)

Re: per-thread global variables

[Robert M. Hyatt]

In the "master thread" I define an array of structures, one per
thread.  When I start a thread, I pass it the address of its private
structure.  Note that this has nothing to do with the stack.  Each
thread _must_ have a stack (mine is 1mb per thread) and you have to
(on intel) pass the _end_ address since stacks go backward on
X86 machines.

Another idea is to set up an array of pointers to the private
(global) structures.  Then have each thread do something
like pointer[thread_id]->stuff to get to stuff in its private
structure.




On Sat, 13 Jul 2002, J.A. Magallon wrote:

> 
> On 2002.07.12 Robert M. Hyatt wrote:
> >
> >
> >I do this in Crafty.  The approach I chose is to create a structure
> >for each thread (I also use clone() since the glibc guys broke 
> >pthreads a while back) and then let each thread set its pointer to
> >its own private structure (really shared, but since only one thread
> >has a pointer to each structure, it is like a private but global
> >group of variables).
> >
> >Doing it like this will guarantee it will work with any compiler,
> >which might be a plus...
> >
> 
> I think something is very dark for me...The hard way in pseudo-C
> 
> short id[PID_MAX];
> #define setself(k) do { tid[getpid()]=k; } while(0)
> #define self() (id[getpid()])
> 
> // master
>   setself(-1);
> // slaves
>   for (i in 0..nslaves-1)
>     clone(f(),CLONE_VM,i);
> 
> f(void *arg)
> {
>   setself((int)arg);
>   print(self());
> }
> 
> But how about the 32K*2 array ??? (hash table, ordered vector and binary
> search...)
> 
> Quoting your answer:
> 
> >pthreads a while back) and then let each thread set its pointer to
>                                                      ^^^
> >its own private structure (really shared, but since only one thread
> 
> How do you define 'its' pointer ? That is like if I just defined
> 
> short id;
> 
> and tried to set 'its id' for each thread ?
> Whichever method you use, at the end you always need one, the last, the
> unique variable (one int, a pointer to a data area), declared as a global
> variable in C that you need to be distinct for each thread. The only
> thing (in my short undestanding) that can distinguish a thread from one
> other is _pid_. Of course, I don't want to mess passing an info struct
> to all functions in my code.
> 
> On IRIX, you have a
> 
> // Fixed address space always private for each process/thread
> #define PRDA        ((struct prda *)0x00200000L)
> 
> struct prda*	thrprda;
> int*	self;
> 
> f(int k)
> {
>   thrprda = (struct prda*)PRDA; // The magic global pointer
>                                 // different for each thread
>   self = &(thrprda->user_area);
>   *self = k;
> }
> 
> If Linux supported this it will make many many things simpler.
> A fixed zone of virtual address space that is never shared.
> 
> Or perhaps... Could I do in Linux:
> 
> #define PRDA        ((void*)0x00200000L) // Or get any free address space
> 
> int* self = PRDA; // will be shared over clone(), so we fix it before
> 
> for (i in 0..nslaves-1)
>   clone(f(),CLONE_VM,i);
> 
> f(void *arg)
> {
>   // Get a chunk, mapped on a fixed address, and do not propagate the
>   // map to parent or siblings
>   mmap(self,1024,PROT_??,MAP_ANONYMOUS|MAP_FIXED|MAP_PRIVATE,0,0);
>   *self = (int)arg;
> }
> 
> 
> ??
> 
> TIA
> 
> 

-- 
Robert Hyatt                    Computer and Information Sciences
hyatt@cis.uab.edu               University of Alabama at Birmingham
(205) 934-2213                  115A Campbell Hall, UAB Station 
(205) 934-5473 FAX              Birmingham, AL 35294-1170

Re: per-thread global variables

[J.A. Magallon]

On 2002.07.13 Robert M. Hyatt wrote:
>
>In the "master thread" I define an array of structures, one per
>thread.  When I start a thread, I pass it the address of its private
>structure.

So:

struct tinfo* data[n];

for (nthreads)
  data[i] = new private struct
  clone(f,data[i])

f(struct tinfo* mydata)
{
  me = mydata->mysefl; // OK till here
  g();
}

g()  // Note I do not pass any tinfo here
{
  // Who am I ?????????
}

You can't call any function from you thread-main function that
needs to self-identify ??

>Note that this has nothing to do with the stack.  Each
>thread _must_ have a stack (mine is 1mb per thread) and you have to
>(on intel) pass the _end_ address since stacks go backward on
>X86 machines.
>
>Another idea is to set up an array of pointers to the private
>(global) structures.  Then have each thread do something
>like pointer[thread_id]->stuff to get to stuff in its private
>structure.
>

We are in the same problem, if thread_id is pid, you need a big array.
If it is not, you need a multi-personality thread_id.

-- 
J.A. Magallon             \   Software is like sex: It's better when it's free
mailto:jamagallon@able.es  \                    -- Linus Torvalds, FSF T-shirt
Linux werewolf 2.4.19-rc1-jam3, Mandrake Linux 8.3 (Cooker) for i586
gcc (GCC) 3.1.1 (Mandrake Linux 8.3 3.1.1-0.7mdk)

Re: per-thread global variables

[Robert M. Hyatt]

On Sat, 13 Jul 2002, J.A. Magallon wrote:

> 
> On 2002.07.13 Robert M. Hyatt wrote:
> >
> >In the "master thread" I define an array of structures, one per
> >thread.  When I start a thread, I pass it the address of its private
> >structure.
> 
> So:
> 
> struct tinfo* data[n];
> 
> for (nthreads)
>   data[i] = new private struct
>   clone(f,data[i])
> 
> f(struct tinfo* mydata)
> {
>   me = mydata->mysefl; // OK till here
>   g();
> }
> 
> g()  // Note I do not pass any tinfo here
> {
>   // Who am I ?????????
> }
> 
> You can't call any function from you thread-main function that
> needs to self-identify ??


pass each new lightweight its ID thru the "arg" facility on the
clone() call.  That is how I do this...



> 
> >Note that this has nothing to do with the stack.  Each
> >thread _must_ have a stack (mine is 1mb per thread) and you have to
> >(on intel) pass the _end_ address since stacks go backward on
> >X86 machines.
> >
> >Another idea is to set up an array of pointers to the private
> >(global) structures.  Then have each thread do something
> >like pointer[thread_id]->stuff to get to stuff in its private
> >structure.
> >
> 
> We are in the same problem, if thread_id is pid, you need a big array.
> If it is not, you need a multi-personality thread_id.
> 
> 

-- 
Robert Hyatt                    Computer and Information Sciences
hyatt@cis.uab.edu               University of Alabama at Birmingham
(205) 934-2213                  115A Campbell Hall, UAB Station 
(205) 934-5473 FAX              Birmingham, AL 35294-1170