mips RDHWR instruction in glibc

mips RDHWR instruction in glibc

[Atsushi Nemoto]

I got many "Reserved Instruction" exceptions with gcc 4.1 + glibc 2.4
userland.  They were due to RDHWR instruction to support TLS.

If a system call returned an error, glibc must save the result to
errno, which is thread-local, so RDHWR used.  I can understand this
scenario.  But it seems the RDHWR is often called on non-error cases.

For example, in the code below, RDHWR is placed _before_ checking the
error.  I suppose these instructions were reordered by gcc's
optimization, but the optimization would have large negative effect in
this case.

00566fc4 <_IO_file_read>:
  566fc4:	3c1c0016 	lui	gp,0x16
  566fc8:	279c87ac 	addiu	gp,gp,-30804
  566fcc:	0399e021 	addu	gp,gp,t9
  566fd0:	8c82003c 	lw	v0,60(a0)
  566fd4:	30420002 	andi	v0,v0,0x2
  566fd8:	14400003 	bnez	v0,566fe8 <_IO_file_read+0x24>
  566fdc:	8f999e9c 	lw	t9,-24932(gp)
  566fe0:	03200008 	jr	t9
  566fe4:	8c840038 	lw	a0,56(a0)
  566fe8:	8c840038 	lw	a0,56(a0)
  566fec:	24020fa3 	li	v0,4003
  566ff0:	0000000c 	syscall
  566ff4:	8f84a528 	lw	a0,-23256(gp)
  566ff8:	7c03e83b 	rdhwr	v1,$29
  566ffc:	00832021 	addu	a0,a0,v1
  567000:	14e00003 	bnez	a3,567010 <_IO_file_read+0x4c>
  567004:	00401821 	move	v1,v0
  567008:	03e00008 	jr	ra
  56700c:	00601021 	move	v0,v1
  567010:	2403ffff 	li	v1,-1
  567014:	1000fffc 	b	567008 <_IO_file_read+0x44>
  567018:	ac820000 	sw	v0,0(a0)

I'm not sure where to fix, but I doubt some inline asm code in glibc
lack "volatile" keyword.

Does anyone have a clue on this?
---
Atsushi Nemoto

Re: mips RDHWR instruction in glibc

[Daniel Jacobowitz]

On Thu, Jun 15, 2006 at 12:12:38AM +0900, Atsushi Nemoto wrote:
> If a system call returned an error, glibc must save the result to
> errno, which is thread-local, so RDHWR used.  I can understand this
> scenario.  But it seems the RDHWR is often called on non-error cases.

Libc uses TLS for many things other than just errno.  The GCC port
knows how to generate the agreed-upon rdhwr instruction directly.

> For example, in the code below, RDHWR is placed _before_ checking the
> error.  I suppose these instructions were reordered by gcc's
> optimization, but the optimization would have large negative effect in
> this case.

You'd have to figure out how to get GCC not to eagerly schedule the
rdhwr.  This might be quite hard.  I don't know much about this part of
the scheduler.

-- 
Daniel Jacobowitz
CodeSourcery

Re: mips RDHWR instruction in glibc

[Atsushi Nemoto]

On Wed, 14 Jun 2006 12:50:40 -0400, Daniel Jacobowitz <dan@debian.org> wrote:
> > For example, in the code below, RDHWR is placed _before_ checking the
> > error.  I suppose these instructions were reordered by gcc's
> > optimization, but the optimization would have large negative effect in
> > this case.
> 
> You'd have to figure out how to get GCC not to eagerly schedule the
> rdhwr.  This might be quite hard.  I don't know much about this part of
> the scheduler.

I really did not understand yet how errno is bound TLS.  I found some
"rdhwr" in glibc-ports source code (tls-macros.h, nptl/tls.h).  The
RDHWR instruction in the example code comes from one of them, no?

I also found a "rdhwr" in gcc's mips.md file ("tls_get_tp_<mode>").
Is this the origin?  MD is a very foreign language for me...

---
Atsushi Nemoto

Re: mips RDHWR instruction in glibc

[Daniel Jacobowitz]

On Fri, Jun 16, 2006 at 12:28:37AM +0900, Atsushi Nemoto wrote:
> On Wed, 14 Jun 2006 12:50:40 -0400, Daniel Jacobowitz <dan@debian.org> wrote:
> > > For example, in the code below, RDHWR is placed _before_ checking the
> > > error.  I suppose these instructions were reordered by gcc's
> > > optimization, but the optimization would have large negative effect in
> > > this case.
> > 
> > You'd have to figure out how to get GCC not to eagerly schedule the
> > rdhwr.  This might be quite hard.  I don't know much about this part of
> > the scheduler.
> 
> I really did not understand yet how errno is bound TLS.  I found some
> "rdhwr" in glibc-ports source code (tls-macros.h, nptl/tls.h).  The
> RDHWR instruction in the example code comes from one of them, no?

No.

> I also found a "rdhwr" in gcc's mips.md file ("tls_get_tp_<mode>").
> Is this the origin?  MD is a very foreign language for me...

Yes.  Compile something like this with -O2 but without -fpic:

__thread int x;
int foo() { return x; }

It should use the IE model, which will generate a rdhwr.

-- 
Daniel Jacobowitz
CodeSourcery

Re: mips RDHWR instruction in glibc

[Atsushi Nemoto]

On Thu, 15 Jun 2006 11:32:52 -0400, Daniel Jacobowitz <dan@debian.org> wrote:
> > I also found a "rdhwr" in gcc's mips.md file ("tls_get_tp_<mode>").
> > Is this the origin?  MD is a very foreign language for me...
> 
> Yes.  Compile something like this with -O2 but without -fpic:
> 
> __thread int x;
> int foo() { return x; }
> 
> It should use the IE model, which will generate a rdhwr.

Thanks.  So this must be a gcc issue, not glibc issue.

extern __thread int x;
int foo(int arg)
{
	if (arg)
		return x;
	return 0;
}

If I compiled this program with -O2 I got:

foo:
	.frame	$sp,0,$31		# vars= 0, regs= 0/0, args= 0, gp= 0
	.mask	0x00000000,0
	.fmask	0x00000000,0
	.set	noreorder
	.cpload	$25
	.set	nomacro
	
	lw	$2,%gottprel(x)($28)
	.set	push
	.set	mips32r2	
	rdhwr	$3,$29
	.set	pop
	addu	$2,$2,$3
	beq	$4,$0,$L4
	move	$3,$0

	lw	$3,0($2)
$L4:
	j	$31
	move	$2,$3

It looks too bad for arg == 0 case.  I should ask on gcc list.

---
Atsushi Nemoto

Re: mips RDHWR instruction in glibc

[Atsushi Nemoto]

On Sat, 17 Jun 2006 00:58:45 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> wrote:
> It looks too bad for arg == 0 case.  I should ask on gcc list.

OTOH, I think it would be better to reduce RDHWR overhead in kernel
anyway.

I've heard about "fastpath" approach by Maciej, but now I can not find
the patch on linux-mips ML archive.  Is it available on somewhere?  Maciej?

---
Atsushi Nemoto

Re: mips RDHWR instruction in glibc

[Atsushi Nemoto]

On Sat, 17 Jun 2006 00:58:45 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> wrote:
> It looks too bad for arg == 0 case.  I should ask on gcc list.

Filed to gcc bugzilla.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=28126

---
Atsushi Nemoto