shmget with SHM_HUGETLB flag: Operation not permitted

shmget with SHM_HUGETLB flag: Operation not permitted

[Jochen Roemling]

Hi,

I'm using stock kernel 2.6.2. I have HUGETLB support compiled in.

CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y

When issuing this command in a C pgm

shmid =shmget(IPC_PRIVATE, SOMESIZE, SHM_HUGETLB|IPC_CREAT|SHM_R|SHM_W)

I get "Operation not Permitted" when running it as a normal user. It
works for root. Without the SHM_HUGETLB flag it works fine for all users.

How can I grant the permission to use HUGETLB to ordinary users?


Jochen

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[William Lee Irwin III]

On Thu, Feb 26, 2004 at 11:36:03PM +0100, Jochen Roemling wrote:
> Hi,
> I'm using stock kernel 2.6.2. I have HUGETLB support compiled in.
> CONFIG_HUGETLBFS=y
> CONFIG_HUGETLB_PAGE=y
> When issuing this command in a C pgm
> shmid =shmget(IPC_PRIVATE, SOMESIZE, SHM_HUGETLB|IPC_CREAT|SHM_R|SHM_W)
> I get "Operation not Permitted" when running it as a normal user. It
> works for root. Without the SHM_HUGETLB flag it works fine for all users.
> How can I grant the permission to use HUGETLB to ordinary users?

(a) use the fs which uses fs permissions to grant users permission to
	fiddle with hugetlb
(b) man 2 capset
(c) proxy daemon for shmget()


-- wli

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[Chris Wright]

* William Lee Irwin III (wli@holomorphy.com) wrote:
> On Thu, Feb 26, 2004 at 11:36:03PM +0100, Jochen Roemling wrote:
> > How can I grant the permission to use HUGETLB to ordinary users?
> 
> (a) use the fs which uses fs permissions to grant users permission to
> 	fiddle with hugetlb
> (b) man 2 capset

In case that part wasn't clear, it would be CAP_IPC_LOCK capability.

thanks,
-chris
-- 
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[Jochen Roemling]

Chris Wright wrote:
> * William Lee Irwin III (wli@holomorphy.com) wrote:
> 
>>On Thu, Feb 26, 2004 at 11:36:03PM +0100, Jochen Roemling wrote:
>>
>>>How can I grant the permission to use HUGETLB to ordinary users?
>>
>>(a) use the fs which uses fs permissions to grant users permission to
>>	fiddle with hugetlb
>>(b) man 2 capset
> 
> 
> In case that part wasn't clear, it would be CAP_IPC_LOCK capability.
> 
Thanks. Capset was the keyword I couldn't remember.

_Background:_
I would like to install Oracle 10g Database on Linux with HUGETLB
support. The oracle binary exits with -EPERM because it is not allowed
to create a shared memory segment with the SHM_HUGETLB flag set.

I installed the libcap2 package (from debian testing) and now have the
tool "setcap" available. I wanted to test this on my example pgm
mentioned in the original post using:

roesrv01~ # setcap CAP_IPC_LOCK a.out
fatal error: Invalid argument
usage: setcap [-q] (-|<caps>) <filename> [ ... (-|<capsN>) <filenameN> ]

using the number "14" instead of the name "CAP_IPC_LOCK" doesn't work
either. I don't have any glue. Do have a simple example for me?

By the way: CAP_IPC_LOCK is only checked in line 508 of ipc/shm.c:

         case SHM_LOCK:
         case SHM_UNLOCK:
         {
/* Allow superuser to lock segment in memory */
/* Should the pages be faulted in here or leave it to user? */
/* need to determine interaction with current->swappable */
                 if (!capable(CAP_IPC_LOCK)) {
                         err = -EPERM;
                         goto out;
                 }

There is nothing around that says: "Allow this only without HUGETLB".
Are you sure that this capability is my problem?

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[Chris Wright]

* Jochen Roemling (jochen@roemling.net) wrote:
> Chris Wright wrote:
> > In case that part wasn't clear, it would be CAP_IPC_LOCK capability.
> > 
> Thanks. Capset was the keyword I couldn't remember.
> 
> _Background:_
> I would like to install Oracle 10g Database on Linux with HUGETLB 
> support. The oracle binary exits with -EPERM because it is not allowed 
> to create a shared memory segment with the SHM_HUGETLB flag set.

OK, as expected.

> I installed the libcap2 package (from debian testing) and now have the 
> tool "setcap" available. I wanted to test this on my example pgm 
> mentioned in the original post using:
> 
> roesrv01~ # setcap CAP_IPC_LOCK a.out
> fatal error: Invalid argument
> usage: setcap [-q] (-|<caps>) <filename> [ ... (-|<capsN>) <filenameN> ]
> 
> using the number "14" instead of the name "CAP_IPC_LOCK" doesn't work 
> either. I don't have any glue. Do have a simple example for me?

did you try setpcaps?  smth like setpcaps cap_ipc_lock+e <pid>

> By the way: CAP_IPC_LOCK is only checked in line 508 of ipc/shm.c:
<snip>
>                  if (!capable(CAP_IPC_LOCK)) {
>                          err = -EPERM;
>                          goto out;
>                  }
> 
> There is nothing around that says: "Allow this only without HUGETLB".
> Are you sure that this capability is my problem?

Yes, take a look at fs/hugetlbfs/inode.c::hugetlb_zero_setup()

thanks,
-chris
-- 
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[Chris Wright]

* Chris Wright (chrisw@osdl.org) wrote:
> did you try setpcaps?  smth like setpcaps cap_ipc_lock+e <pid>

bah, sorry, i should point out, that isn't going to work w/out CAP_SETPCAP
which is disabled.  you'll want to start with full privs (i.e. root) and
drop all but CAP_IPC_LOCK.  SuSE used to have a tool called compartment
that helped with this, might google for it.

thanks,
-chris
-- 
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[Jochen Roemling]

Chris Wright wrote:
> did you try setpcaps?  smth like setpcaps cap_ipc_lock+e <pid>

Ok. One step further now. The syntax seems correct now. I tried to grant 
capabilities to the user's shell:

roesrv01~ # setpcaps cap_ipc_lock+e 2864
[caps set to:
= cap_ipc_lock+e
]
Failed to set cap's on process `2864': (Operation not permitted)

and with the setcap tool again:

roesrv01~ # setcap cap_ipc_lock+e a.out
Failed to set capabilities on file `a.out'
  (Function not implemented)

Hmmm. What do we do now?

>>Are you sure that this capability is my problem?
> 
> Yes, take a look at fs/hugetlbfs/inode.c::hugetlb_zero_setup()

Ok, this would explain it. But what role plays the pseudo-filesystem in 
this case? I don't have it mounted. Isn't it only needed when using 
mmap, not shmget? I guess, I have a serious lack of knownledge here.

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[Wim Coekaerts]

On Thu, Feb 26, 2004 at 04:06:16PM -0800, Chris Wright wrote:
> * Jochen Roemling (jochen@roemling.net) wrote:
> > Chris Wright wrote:
<snip> 
> >                  if (!capable(CAP_IPC_LOCK)) {
> >                          err = -EPERM;
> >                          goto out;
> >                  }
> > 
> > There is nothing around that says: "Allow this only without HUGETLB".
> > Are you sure that this capability is my problem?
> 
> Yes, take a look at fs/hugetlbfs/inode.c::hugetlb_zero_setup()

Rik had a patch in rhel3 for nonroot mlock() which made this all work,
I will post a patch for 2.6. from what I understand it's very useful for
the gpg folks to have, eg wide audience for this patch. just need to do
a bit more testing and having Rik review it then ll send it out (or
maybe he feels like getting shot and he will ;)

Wim

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[Jochen Roemling]

Chris Wright wrote:

>SuSE used to have a tool called compartment
>that helped with this, might google for it.
>  
>
sounds good, but does not work either :-(

roesrv01~ # compartment --cap CAP_IPC_LOCK bash
bash-2.05b# /sbin/getpcaps 3226
Capabilities for `3226': =ep cap_ipc_lock+i cap_setfcap-p cap_setpcap-ep
bash-2.05b# su - jochen
jochen@roesrv01:~> /sbin/getpcaps 3233
Capabilities for `3233': = cap_ipc_lock+i
jochen@roesrv01:~> ./a.out
Failure:: Operation not permitted
jochen@roesrv01:~> ps ax
[...]
 3226 pts/0    S      0:00 bash
 3233 pts/0    S      0:00 -su

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[William Lee Irwin III]

On Fri, Feb 27, 2004 at 01:35:41AM +0100, Jochen Roemling wrote:
> Ok. One step further now. The syntax seems correct now. I tried to grant 
> capabilities to the user's shell:
> roesrv01~ # setpcaps cap_ipc_lock+e 2864
> [caps set to:
> = cap_ipc_lock+e
> ]
> Failed to set cap's on process `2864': (Operation not permitted)
> and with the setcap tool again:

This is likely due to not having the capability to grant in the granting
process. Things are supposed to be vauely montonic here.

On Fri, Feb 27, 2004 at 01:35:41AM +0100, Jochen Roemling wrote:
> roesrv01~ # setcap cap_ipc_lock+e a.out
> Failed to set capabilities on file `a.out'
>  (Function not implemented)
> Hmmm. What do we do now?

setcap on executables probably isn't supported by your fs.


-- wli

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[William Lee Irwin III]

On Fri, Feb 27, 2004 at 01:55:39AM +0100, Jochen Roemling wrote:
> sounds good, but does not work either :-(
> roesrv01~ # compartment --cap CAP_IPC_LOCK bash
> bash-2.05b# /sbin/getpcaps 3226
> Capabilities for `3226': =ep cap_ipc_lock+i cap_setfcap-p cap_setpcap-ep
> bash-2.05b# su - jochen
> jochen@roesrv01:~> /sbin/getpcaps 3233
> Capabilities for `3233': = cap_ipc_lock+i
> jochen@roesrv01:~> ./a.out
> Failure:: Operation not permitted
> jochen@roesrv01:~> ps ax
> [...]
> 3226 pts/0    S      0:00 bash
> 3233 pts/0    S      0:00 -su

Check /proc/sys/vm/nr_hugepages and /proc/sys/kernel/shmmax also.


-- wli

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[Jochen Roemling]

William Lee Irwin III wrote:

>Check /proc/sys/vm/nr_hugepages and /proc/sys/kernel/shmmax also.
>  
>
cat /proc/sys/vm/nr_hugepages
64

cat /proc/sys/kernel/shmmax
33554432

cat /proc/meminfo | grep Huge
HugePages_Total:    64
HugePages_Free:     62
Hugepagesize:     4096 kB

but again: root can, users cannot, so sizes won't matter, would they?

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[William Lee Irwin III]

William Lee Irwin III wrote:
>>Check /proc/sys/vm/nr_hugepages and /proc/sys/kernel/shmmax also.

On Fri, Feb 27, 2004 at 02:33:08AM +0100, Jochen Roemling wrote:
> cat /proc/sys/vm/nr_hugepages
> 64

256MB limit there.


On Fri, Feb 27, 2004 at 02:33:08AM +0100, Jochen Roemling wrote:
> cat /proc/sys/kernel/shmmax
> 33554432

32MB limit there.


On Fri, Feb 27, 2004 at 02:33:08AM +0100, Jochen Roemling wrote:
> cat /proc/meminfo | grep Huge
> HugePages_Total:    64
> HugePages_Free:     62
> Hugepagesize:     4096 kB
> but again: root can, users cannot, so sizes won't matter, would they?

It's capable(CAP_IPC_LOCK) || in_group_p(0), not current->uid == 0.
It will barf if you ask for more than either one of those limits. It
will also barf if you ask for an amount not a multiple of the hugepage
size. Please show the test program's code and strace the test program
to determine what response it's getting.


-- wli

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[Zlatko Calusic]

Jochen Roemling <jochen@roemling.net> writes:

> William Lee Irwin III wrote:
>
>>Check /proc/sys/vm/nr_hugepages and /proc/sys/kernel/shmmax also.
>>
>>
> cat /proc/sys/vm/nr_hugepages
> 64
>
> cat /proc/sys/kernel/shmmax
> 33554432
>
> cat /proc/meminfo | grep Huge
> HugePages_Total:    64
> HugePages_Free:     62
> Hugepagesize:     4096 kB
>
> but again: root can, users cannot, so sizes won't matter, would they?

Of course! Appended simple patch is what i did (ugly, I know) and that
helped me install Oracle10g on Debian unstable (with two other
adaptations). I don't know how in the hell I forgot to put that
important patch on my page where I explain how to install Oracle10g on
Debian?! Sorry, it'll be on http://linux.inet.hr/oracle10g_on_debian.html
later today or tomorrow, after I check some other problems people have
reported to me (and you Jochen, too :)).

Index: 3.3/fs/hugetlbfs/inode.c
--- 3.3/fs/hugetlbfs/inode.c Thu, 19 Feb 2004 19:05:15 +0100 zcalusic (linux26/D/6_inode.c 1.1.1.2 644)
+++ 3.4/fs/hugetlbfs/inode.c Mon, 23 Feb 2004 09:33:52 +0100 zcalusic (linux26/D/6_inode.c 1.1.1.3 644)
@@ -694,9 +694,6 @@
 	struct qstr quick_string;
 	char buf[16];
 
-	if (!capable(CAP_IPC_LOCK))
-		return ERR_PTR(-EPERM);
-
 	if (!is_hugepage_mem_enough(size))
 		return ERR_PTR(-ENOMEM);
 	n = atomic_read(&hugetlbfs_counter);

Regards,
-- 
Zlatko

P.S. Please Cc: me, I'm not subscribed.

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[William Lee Irwin III]

On Fri, Feb 27, 2004 at 05:32:46PM +0100, Zlatko Calusic wrote:
> Of course! Appended simple patch is what i did (ugly, I know) and that
> helped me install Oracle10g on Debian unstable (with two other
> adaptations). I don't know how in the hell I forgot to put that
> important patch on my page where I explain how to install Oracle10g on
> Debian?! Sorry, it'll be on http://linux.inet.hr/oracle10g_on_debian.html
> later today or tomorrow, after I check some other problems people have
> reported to me (and you Jochen, too :)).

You have to be a bit more careful than this; this gives any user the
ability to consume locked memory via shmget().


-- wli

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[Zlatko Calusic]

William Lee Irwin III <wli@holomorphy.com> writes:

> On Fri, Feb 27, 2004 at 05:32:46PM +0100, Zlatko Calusic wrote:
>> Of course! Appended simple patch is what i did (ugly, I know) and that
>> helped me install Oracle10g on Debian unstable (with two other
>> adaptations). I don't know how in the hell I forgot to put that
>> important patch on my page where I explain how to install Oracle10g on
>> Debian?! Sorry, it'll be on http://linux.inet.hr/oracle10g_on_debian.html
>> later today or tomorrow, after I check some other problems people have
>> reported to me (and you Jochen, too :)).
>
> You have to be a bit more careful than this; this gives any user the
> ability to consume locked memory via shmget().

Yes, I know! But hopefully this security implication is not so
important for people who just want to test new database on their
workstations (like me), or even install it on the production database
server where you mostly don't see any other shell user beside the
administrator of the machine.

But yes, you're right, we need to warn people.

DON'T use the patch if you have untrusty shell users on your machine!!!

-- 
Zlatko

P.S. Although, if superuser properly limits the number of hugepages
     that can be allocated (echo "valid nr of pages" > nr_hugepages)
     what does attacker do to consume more pages than that? Just
     curious...

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[Jochen Roemling]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]

William Lee Irwin III wrote:

>It's capable(CAP_IPC_LOCK) || in_group_p(0), not current->uid == 0.
>It will barf if you ask for more than either one of those limits. It
>will also barf if you ask for an amount not a multiple of the hugepage
>size. Please show the test program's code and strace the test program
>to determine what response it's getting.
>
>  
>
I attached the test pgm. It is nearly the same as shown
in Documentation/vm/hugetlbpage.txt

If you run it as root, it allocates 1 Hugepage, if run as user, it fails.

roesrv01:~ # ./a.out
shmid: 0x220004
shmaddr: 0x40167000
Starting the writes:
....
Starting the Check...Done.
roesrv01:~ # su - jochen
jochen@roesrv01:~> ./a.out
Failure:: Operation not permitted

I guess, a strace is not necessary.
The pgm has only the main function and only one position where it says 
"Failure"

What do I have to do to make this pgm run as an ordinary user with a 
stock kernel?

Curious...
Jochen

[-- Attachment #2: huge.c --]
[-- Type: text/plain, Size: 1799 bytes --]

/* Example of using hugepage in user application using Sys V shared memory
 * system calls.  In this example, app is requesting memory of size 256MB that
 * is backed by huge pages.  Application uses the flag SHM_HUGETLB in shmget
 * system call to informt the kernel that it is requesting hugepages.  For
 * IA-64 architecture, Linux kernel reserves Region number 4 for hugepages.
 * That means the addresses starting with 0x800000....will need to be
 * specified.
 */
#include <sys/types.h>
#include <sys/shm.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <errno.h>

extern int errno;
#define SHM_HUGETLB 04000
#define LPAGE_SIZE      (4UL*1024UL*1024UL)
#define         dprintf(x)  printf(x)
#define ADDR (0x8000000000000000UL)
main()
{
        int shmid;
        int     i, j, k;
        volatile        char    *shmaddr;

	if ((shmid =shmget(IPC_PRIVATE, LPAGE_SIZE, SHM_HUGETLB|IPC_CREAT|SHM_R|SHM_W )) < 0) {
                perror("Failure:");
                exit(1);
        }
        printf("shmid: 0x%x\n", shmid);
        shmaddr = shmat(shmid, (void *)ADDR, SHM_RND) ;
        if (errno != 0) {
                perror("Shared Memory Attach Failure:");
                exit(2);
        }
        printf("shmaddr: %p\n", shmaddr);

        dprintf("Starting the writes:\n");
        for (i=0;i<LPAGE_SIZE;i++) {
                shmaddr[i] = (char) (i);
                if (!(i%(1024*1024))) dprintf(".");
        }
        dprintf("\n");
        dprintf("Starting the Check...");
        for (i=0; i<LPAGE_SIZE;i++)
                if (shmaddr[i] != (char)i)
                        printf("\nIndex %d mismatched.");
        dprintf("Done.\n");
        if (shmdt((const void *)shmaddr) != 0) {
                perror("Detached Failure:");
                exit (3);
        }
}

Re: shmget with SHM_HUGETLB flag: Operation not permitted

[William Lee Irwin III]

William Lee Irwin III wrote:
>> It's capable(CAP_IPC_LOCK) || in_group_p(0), not current->uid == 0.
>> It will barf if you ask for more than either one of those limits. It
>> will also barf if you ask for an amount not a multiple of the hugepage
>> size. Please show the test program's code and strace the test program
>> to determine what response it's getting.

On Sun, Feb 29, 2004 at 10:37:38PM +0100, Jochen Roemling wrote:
> What do I have to do to make this pgm run as an ordinary user with a 
> stock kernel?

Locked memory is a privileged resource, so you do have to do something
to authenticate lest any user be able to consume all memory on your
system with no possibility of paging it. Examples of what to do to
acquire locked memory specifically for hugetlb shm segments in mainline:

(a) give the user gid 0 as a primary or supplementary group
(b) grant the capability -- yes, it can be done (and is being done in
	practice elsewhere), something is going wrong on your end I
	haven't been able to diagnose.
(c) make requests from a shmget() proxy daemon where you make requests
	over a socket and it hands back shm segment ID's that have had
	their uid's/perms set to the end user. Once shmget() is done,
	shmat() uses normal shm permissions checks.
(d) use a setuid root shmget() helper app.
(e) launch as root, then retain capabilities
(f) launch as root and shmget before dropping privs

(e) and (f) are probably not options in your case. I can't predict
what's going to be desirable on your end in general. You will have to
jump through a hoop of some kind, though, and be glad you do, since
otherwise unbounded amounts of locked memory requested by arbitrary
users could cripple the system's performance or worse.


-- wli