System Call trashing registers

System Call trashing registers

[Joshua Hudson]

1) A lot of my old 32-bit programs don't work on x64 linux anymore
because int 80h now trashes ecx and edx. This hasn't been a serious
problem for me.

2) syscall is documented to trash rcx and r11.

What I don't understand is why this hasn't ever led to a security
issue due to leaking values from kernel space (in the trashed
registers) back to userspace.

Re: System Call trashing registers

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

Hi!

> 1) A lot of my old 32-bit programs don't work on x64 linux anymore
> because int 80h now trashes ecx and edx. This hasn't been a serious
> problem for me.
> 
> 2) syscall is documented to trash rcx and r11.
> 
> What I don't understand is why this hasn't ever led to a security
> issue due to leaking values from kernel space (in the trashed
> registers) back to userspace.

I guess you should cc x86 maintainers?

I guess this is a regression? When did it start?

BR,
								Pavel
-- 
People of Russia, stop Putin before his war on Ukraine escalates.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: System Call trashing registers

[Joshua Hudson]

On Mon, Aug 28, 2023 at 8:06 AM Pavel Machek <pavel@ucw.cz> wrote:
>
> Hi!
>
> > 1) A lot of my old 32-bit programs don't work on x64 linux anymore
> > because int 80h now trashes ecx and edx. This hasn't been a serious
> > problem for me.
> >
>
> I guess you should cc x86 maintainers?
>
> I guess this is a regression? When did it start?
>
> BR,
>                                                                 Pavel
> --
> People of Russia, stop Putin before his war on Ukraine escalates.

It started when the openat() syscall was added. It got fixed a few times.
I suspect it stayed fixed on an actual x86 kernel but the x86_64 kernel
still exhibits it being broken. But I don't run an x86 kernel anymore for
the obvious reason.

Source code can be found here. The project effectively died with
floppy disks so last release in 2006 is about right.

https://asm.sourceforge.net/asmutils.html

But if I try the couple of tools that still have real value, they just
segfault.

Remaining useful tools: tr, httpd, chmod, mv, ln

RE: System Call trashing registers

[David Laight]

From: Joshua Hudson
> Sent: 24 August 2023 17:15
> 
> 1) A lot of my old 32-bit programs don't work on x64 linux anymore
> because int 80h now trashes ecx and edx. This hasn't been a serious
> problem for me.

Aren't both ecx and edx caller saved?
So if the code is using asm syscall wrappers provided the asm
wrappers don't expect the registers be saved (which is unlikely)
then it is safe for the kernel to trash them.

OTOH I have seen code compiled with inlined syscall code
(but not recently). In that case it will matter.

It gets more interesting with all the xmm/ymm/zmm registers.
They are also all caller saved, so if the compiler always
sees a real function call wrapping a system call then the kernel
need not save them and can return with them all set to zero.

I don't believe that is done, but it is likely to be a measurable
performance gain for most programs.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: System Call trashing registers

[Joshua Hudson]

On Mon, Aug 28, 2023 at 10:06 AM David Laight <David.Laight@aculab.com> wrote:
>
> From: Joshua Hudson
> > Sent: 24 August 2023 17:15
> >
> > 1) A lot of my old 32-bit programs don't work on x64 linux anymore
> > because int 80h now trashes ecx and edx. This hasn't been a serious
> > problem for me.
>
> Aren't both ecx and edx caller saved?
> So if the code is using asm syscall wrappers provided the asm
> wrappers don't expect the registers be saved (which is unlikely)
> then it is safe for the kernel to trash them.
>
> OTOH I have seen code compiled with inlined syscall code
> (but not recently). In that case it will matter.
>
> It gets more interesting with all the xmm/ymm/zmm registers.
> They are also all caller saved, so if the compiler always
> sees a real function call wrapping a system call then the kernel
> need not save them and can return with them all set to zero.
>
> I don't believe that is done, but it is likely to be a measurable
> performance gain for most programs.
>
>         David
>
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
> Registration No: 1397386 (Wales)

The actual question is "why isn't trashing the registers with arbitrary values
from the kernel stack" a security vulnerability? It doesn't look like
they're being
set to zero.

It was in fact once documented that all registers are preserved except eax.
(Fun fact: this used to be true across a successful exec(); you could actually
pass extra values to a child exec process in esi and edi until this was
deliberately changed to zero all registers on program entry.)

Re: System Call trashing registers

[Ammar Faizi]

On 8/24/23 11:15 PM, Joshua Hudson wrote:
> 1) A lot of my old 32-bit programs don't work on x64 linux anymore
> because int 80h now trashes ecx and edx. This hasn't been a serious
> problem for me.

Do you have a reproducer? It doesn't trash ecx and edx on my machine.

Linux 6.5.0-rc5-af-home-2023-08-08-gf01d31303231
```
#include <stdio.h>

static void do_int80(void)
{
	int ecx = 0x11111;
	int edx = 0x22222;
	int eax = 158; // sched_yield

	__asm__ volatile (
		"int $0x80"
		: "+a"(eax), "+c"(ecx), "+d"(edx)
		:
		: "memory"
	);
	printf("ecx = %#x\n", ecx);
	printf("edx = %#x\n", edx);
}

int main(void)
{
	int i;

	for (i = 0; i < 3; i++)
		do_int80();

	return 0;
}
```

ammarfaizi2@integral2:/tmp$ gcc -Wall -Wextra -Os z.c -o z
ammarfaizi2@integral2:/tmp$ ./z
ecx = 0x11111
edx = 0x22222
ecx = 0x11111
edx = 0x22222
ecx = 0x11111
edx = 0x22222


> 2) syscall is documented to trash rcx and r11.
> 
> What I don't understand is why this hasn't ever led to a security
> issue due to leaking values from kernel space (in the trashed
> registers) back to userspace.

That behavior is architectural. It's the 'syscall' instruction that
clobbers %rcx and %r11. Not the kernel.

The kernel's syscall entry point even saves %rcx and %r11, but at that
point they've already been overwritten by the syscall instruction
itself with the original %rip and %rflags values. So they contain
userspace values. No internal kernel data is leaked in %rcx and %r11.

-- 
Ammar Faizi

Re: System Call trashing registers

[Thomas Gleixner]

On Fri, Sep 01 2023 at 23:24, Ammar Faizi wrote:
> On 8/24/23 11:15 PM, Joshua Hudson wrote:
>> 2) syscall is documented to trash rcx and r11.
>> 
>> What I don't understand is why this hasn't ever led to a security
>> issue due to leaking values from kernel space (in the trashed
>> registers) back to userspace.
>
> That behavior is architectural. It's the 'syscall' instruction that
> clobbers %rcx and %r11. Not the kernel.
>
> The kernel's syscall entry point even saves %rcx and %r11, but at that
> point they've already been overwritten by the syscall instruction
> itself with the original %rip and %rflags values. So they contain
> userspace values. No internal kernel data is leaked in %rcx and %r11.

Correct.

It does not matter which entry method you use. The kernel always saves
all registers and restores them. syscall, sysenter, int80 behave the
same way. The implicit clobber by the syscall instruction is done in
hardware and the kernel can't do anything about it.

I can't reproduce this either and the code tells me that any attempt to
reproduce is futile.

Thanks,

        tglx

Re: System Call trashing registers

[Joshua Hudson]

On Fri, Sep 1, 2023 at 9:24 AM Ammar Faizi <ammarfaizi2@gnuweeb.org> wrote:
>
> On 8/24/23 11:15 PM, Joshua Hudson wrote:
> > 1) A lot of my old 32-bit programs don't work on x64 linux anymore
> > because int 80h now trashes ecx and edx. This hasn't been a serious
> > problem for me.
>
> Do you have a reproducer? It doesn't trash ecx and edx on my machine.
>
> Linux 6.5.0-rc5-af-home-2023-08-08-gf01d31303231
> ```
> #include <stdio.h>
>
> static void do_int80(void)
> {
>         int ecx = 0x11111;
>         int edx = 0x22222;
>         int eax = 158; // sched_yield
>
>         __asm__ volatile (
>                 "int $0x80"
>                 : "+a"(eax), "+c"(ecx), "+d"(edx)
>                 :
>                 : "memory"
>         );
>         printf("ecx = %#x\n", ecx);
>         printf("edx = %#x\n", edx);
> }
>
> int main(void)
> {
>         int i;
>
>         for (i = 0; i < 3; i++)
>                 do_int80();
>
>         return 0;
> }
> ```
>
> ammarfaizi2@integral2:/tmp$ gcc -Wall -Wextra -Os z.c -o z
> ammarfaizi2@integral2:/tmp$ ./z
> ecx = 0x11111
> edx = 0x22222
> ecx = 0x11111
> edx = 0x22222
> ecx = 0x11111
> edx = 0x22222
>
>
> > 2) syscall is documented to trash rcx and r11.
> >
> > What I don't understand is why this hasn't ever led to a security
> > issue due to leaking values from kernel space (in the trashed
> > registers) back to userspace.
>
> That behavior is architectural. It's the 'syscall' instruction that
> clobbers %rcx and %r11. Not the kernel.
>
> --
> Ammar Faizi

Correction: it's been fixed again. Sorry about that.

I know the asmutils tools have been broken for a decade, but they're
working now.

What would happen is system calls that take arguments in ecx and edx would find
ecx and edx trashed, but only a few calls actually did this, the
primary offender
being open(). The best regression test seems to be hexdump from
asmutils because the
corruption would reliably crash the binary. sched_yeild was never an
affected syscall.

Re: System Call trashing registers

[Thomas Gleixner]

On Fri, Sep 01 2023 at 10:38, Joshua Hudson wrote:
> On Fri, Sep 1, 2023 at 9:24 AM Ammar Faizi <ammarfaizi2@gnuweeb.org> wrote:
>
> Correction: it's been fixed again. Sorry about that.

When exactly was it broken? Which kernel version?

The save/restore mechanism is not changing every other day.

> I know the asmutils tools have been broken for a decade, but they're
> working now.
>
> What would happen is system calls that take arguments in ecx and edx would find
> ecx and edx trashed, but only a few calls actually did this, the
> primary offender
> being open(). The best regression test seems to be hexdump from
> asmutils because the
> corruption would reliably crash the binary. sched_yeild was never an
> affected syscall.

It does absolutely not matter which syscall is used. The save/restore
mechanism is identical for all syscalls.

Thanks,

        tglx