[PATCH] Fix compat shmget overflow

[PATCH] Fix compat shmget overflow

[Andi Kleen]

This fixes an incorrect sign extension in the compat layer that 
breaks 32bit shmget that are >2GB. sys_shmget has a signed size_t size
argument, and the int size argument comming from 32bit user space would get 
sign extended to 64bit, which is wrong.

I fixed it on all compat architectures, except PPC64 which was already
ok. 

It was originally debugged and fixed by Karl Rister @ IBM for SLES9 on x86-64.

Signed-off-by: Andi Kleen <ak@suse.de>


diff -u linux-2.6.11rc3/arch/sparc64/kernel/sys_sparc32.c-X linux-2.6.11rc3/arch/sparc64/kernel/sys_sparc32.c
--- linux-2.6.11rc3/arch/sparc64/kernel/sys_sparc32.c-X	2005-02-04 09:12:50.000000000 +0100
+++ linux-2.6.11rc3/arch/sparc64/kernel/sys_sparc32.c	2005-02-09 10:39:52.000000000 +0100
@@ -835,7 +835,7 @@
 			err = sys_shmdt(ptr);
 			goto out;
 		case SHMGET:
-			err = sys_shmget(first, second, third);
+			err = sys_shmget(first, (unsigned)second, third);
 			goto out;
 		case SHMCTL:
 			err = do_sys32_shmctl(first, second, ptr);
diff -u linux-2.6.11rc3/arch/ia64/ia32/sys_ia32.c-X linux-2.6.11rc3/arch/ia64/ia32/sys_ia32.c
--- linux-2.6.11rc3/arch/ia64/ia32/sys_ia32.c-X	2005-02-04 09:12:42.000000000 +0100
+++ linux-2.6.11rc3/arch/ia64/ia32/sys_ia32.c	2005-02-09 10:39:03.000000000 +0100
@@ -1415,7 +1415,7 @@
 	      case SHMDT:
 		return sys_shmdt(compat_ptr(ptr));
 	      case SHMGET:
-		return sys_shmget(first, second, third);
+		return sys_shmget(first, (unsigned)second, third);
 	      case SHMCTL:
 		return compat_sys_shmctl(first, second, compat_ptr(ptr));
 
diff -u linux-2.6.11rc3/arch/mips/kernel/linux32.c-X linux-2.6.11rc3/arch/mips/kernel/linux32.c
--- linux-2.6.11rc3/arch/mips/kernel/linux32.c-X	2005-02-04 09:12:45.000000000 +0100
+++ linux-2.6.11rc3/arch/mips/kernel/linux32.c	2005-02-09 10:39:03.000000000 +0100
@@ -1115,7 +1115,7 @@
 		err = sys_shmdt ((char *)A(ptr));
 		break;
 	case SHMGET:
-		err = sys_shmget (first, second, third);
+		err = sys_shmget (first, (unsigned)second, third);
 		break;
 	case SHMCTL:
 		err = do_sys32_shmctl (first, second, (void *)AA(ptr));
diff -u linux-2.6.11rc3/arch/x86_64/ia32/ipc32.c-X linux-2.6.11rc3/arch/x86_64/ia32/ipc32.c
--- linux-2.6.11rc3/arch/x86_64/ia32/ipc32.c-X	2004-04-06 13:12:04.000000000 +0200
+++ linux-2.6.11rc3/arch/x86_64/ia32/ipc32.c	2005-02-09 10:39:03.000000000 +0100
@@ -49,7 +49,7 @@
 	      case SHMDT:
 		return sys_shmdt(compat_ptr(ptr));
 	      case SHMGET:
-		return sys_shmget(first, second, third);
+		return sys_shmget(first, (unsigned)second, third);
 	      case SHMCTL:
 		return compat_sys_shmctl(first, second, compat_ptr(ptr));
 	}
diff -u linux-2.6.11rc3/arch/s390/kernel/compat_linux.c-X linux-2.6.11rc3/arch/s390/kernel/compat_linux.c
--- linux-2.6.11rc3/arch/s390/kernel/compat_linux.c-X	2005-02-04 09:12:49.000000000 +0100
+++ linux-2.6.11rc3/arch/s390/kernel/compat_linux.c	2005-02-09 10:39:28.000000000 +0100
@@ -331,7 +331,7 @@
 	case SHMDT:
 		return sys_shmdt(compat_ptr(ptr));
 	case SHMGET:
-		return sys_shmget(first, second, third);
+		return sys_shmget(first, (unsigned)second, third);
 	case SHMCTL:
 		return compat_sys_shmctl(first, second, compat_ptr(ptr));
 	}

Re: [PATCH] Fix compat shmget overflow

[Martin Schwidefsky]

> This fixes an incorrect sign extension in the compat layer that
> breaks 32bit shmget that are >2GB. sys_shmget has a signed size_t size
> argument, and the int size argument comming from 32bit user space would
get
> sign extended to 64bit, which is wrong.

Ok for s390 though not needed. The address space for a 31 bit programs
is limited to 2GB anyway.

As you mention sys_shmget, I'm currently trying to find out why the
second argument of sys_ipc for ppc64, s390-64 and sparc64 has been
changed from "long" to "int". This limits the maximum size of a shared
memory segment to 2GB for the three archs. A cast to unsigned would
allow 4GB, still not enough for a 64 bit architecture in the long run.
Before I submit a patch to replace "int" with "long" again I'd like
to understand the reason for the change.

blue skies,
   Martin

Martin Schwidefsky
Linux for zSeries Development & Services
IBM Deutschland Entwicklung GmbH

Re: [PATCH] Fix compat shmget overflow

[Andi Kleen]

> As you mention sys_shmget, I'm currently trying to find out why the
> second argument of sys_ipc for ppc64, s390-64 and sparc64 has been
> changed from "long" to "int". This limits the maximum size of a shared
> memory segment to 2GB for the three archs. A cast to unsigned would
> allow 4GB, still not enough for a 64 bit architecture in the long run.
> Before I submit a patch to replace "int" with "long" again I'd like
> to understand the reason for the change.

I don't know. Look up the Cset at linux.bkbits.net and if it's not clear
from the description ask the submitter? 

-Andi

Re: [PATCH] Fix compat shmget overflow

[Martin Schwidefsky]

Andi Kleen <ak@suse.de> wrote on 09.02.2005 11:31:39:

> > As you mention sys_shmget, I'm currently trying to find out why the
> > second argument of sys_ipc for ppc64, s390-64 and sparc64 has been
> > changed from "long" to "int". This limits the maximum size of a shared
> > memory segment to 2GB for the three archs. A cast to unsigned would
> > allow 4GB, still not enough for a 64 bit architecture in the long run.
> > Before I submit a patch to replace "int" with "long" again I'd like
> > to understand the reason for the change.
>
> I don't know. Look up the Cset at linux.bkbits.net and if it's not clear
> from the description ask the submitter?

Wasn't that easy because arch/s390x/.. isn't under revision control anymore.
Everything in arch/s390x and include/asm-s390x has been removed with 2.5.68.
This is at the same time the revision the shmget bug has been introduced to
s390-64. The second parameter of sys_ipc has been a "long" in the s390x version
of sys_s390.c until s390x got merged into s390 with the step from 2.5.67
to 2.5.68.
That explains it for s390-64 and I'll fix this by changing "int second" to
"unsigned long second" and adding a few casts. While I'm at it I'll fix this
for ppc64 and sparc64 as well.

blue skies,
   Martin

Martin Schwidefsky
Linux for zSeries Development & Services
IBM Deutschland Entwicklung GmbH

Re: [PATCH] Fix compat shmget overflow

[Ralf Baechle]

On Wed, Feb 09, 2005 at 11:08:54AM +0100, Martin Schwidefsky wrote:

> > This fixes an incorrect sign extension in the compat layer that
> > breaks 32bit shmget that are >2GB. sys_shmget has a signed size_t size
> > argument, and the int size argument comming from 32bit user space would
> get
> > sign extended to 64bit, which is wrong.
> 
> Ok for s390 though not needed. The address space for a 31 bit programs
> is limited to 2GB anyway.

Same on MIPS.

  Ralf