Re: [stable] 2.6.16.6 breaks java... sort of

Re: [stable] 2.6.16.6 breaks java... sort of

[Greg KH]

On Wed, Apr 19, 2006 at 12:52:33PM -0600, David Wilk wrote:
> Howdy folks.  thanks for the great work on the stable series, btw.
> 
> I think an issue was introduced with mprotect (the first patch in
> 2.6.16.6).  With 2.6.16.5, tomcat runs fine (in sun-1.5), but in
> 2.6.16.7, the JVM bails out complaining that it couldn't allocate
> enough heap space.
> 
> If I remove '-Xmx768m' from JAVA_OPTS, then the JVM is able to
> startup.  The machine had 1GB of RAM and 2GB of swap, so it should
> have had plenty to give the JVM the 1GB it expects to get with an Xmx
> of 768MB, and this worked in 2.6.16.5 and below.
> 
> I don't know if this is expected and satisfactory behavior, but I
> figured I should give ya'll the heads up.

Odds are it isn't "expected", but Hugh would be the best judge of that.
Hugh?

thanks for letting us know about this,

greg k-h

Re: [stable] 2.6.16.6 breaks java... sort of

[Hugh Dickins]

On Wed, 19 Apr 2006, Greg KH wrote:
> On Wed, Apr 19, 2006 at 12:52:33PM -0600, David Wilk wrote:
> > 
> > I think an issue was introduced with mprotect (the first patch in
> > 2.6.16.6).  With 2.6.16.5, tomcat runs fine (in sun-1.5), but in
> > 2.6.16.7, the JVM bails out complaining that it couldn't allocate
> > enough heap space.
> > 
> > If I remove '-Xmx768m' from JAVA_OPTS, then the JVM is able to
> > startup.  The machine had 1GB of RAM and 2GB of swap, so it should
> > have had plenty to give the JVM the 1GB it expects to get with an Xmx
> > of 768MB, and this worked in 2.6.16.5 and below.
> > 
> > I don't know if this is expected and satisfactory behavior, but I
> > figured I should give ya'll the heads up.
> 
> Odds are it isn't "expected", but Hugh would be the best judge of that.
> Hugh?

Neither expected nor satisfactory.  Sorry about that.  We were hoping
the straightforward shm/mprotect fix would be good enough, but it
appears not.  JVM is probably doing something we can allow with a
more complicated patch, but it _might_ turn out to be doing something
we simply cannot allow: I'll hope for the first and work out a patch
for that; but won't be ready to post it until tomorrow.

If you can get an strace of what it's doing, David, please mail that
to me; but I'm guessing that it may be hard to get, and dispiritingly
large: so don't worry unless it's easy for you.

> thanks for letting us know about this,

Indeed, and sorry for the inconvenience.

Hugh

Re: [stable] 2.6.16.6 breaks java... sort of

[Hugh Dickins]

On Wed, 19 Apr 2006, Hugh Dickins wrote:
> On Wed, 19 Apr 2006, Greg KH wrote:
> > On Wed, Apr 19, 2006 at 12:52:33PM -0600, David Wilk wrote:
> > > 
> > > I think an issue was introduced with mprotect (the first patch in
> > > 2.6.16.6).  With 2.6.16.5, tomcat runs fine (in sun-1.5), but in
> > > 2.6.16.7, the JVM bails out complaining that it couldn't allocate
> > > enough heap space.
> 
> Neither expected nor satisfactory.  Sorry about that.  We were hoping
> the straightforward shm/mprotect fix would be good enough, but it
> appears not.  JVM is probably doing something we can allow with a
> more complicated patch, but it _might_ turn out to be doing something
> we simply cannot allow: I'll hope for the first and work out a patch
> for that; but won't be ready to post it until tomorrow.

David, would you please try this patch on top of your 2.6.16.7 or later.
The first hunk undoes the problematic patch, the remainder does it in a
more permissive way.  Aesthetically, not as satisfactory as the previous
patch; but it's important that we not break userspace, unless security
absolutely demands.  Please let us know how this fares: thanks.

Hugh

--- 2.6.16.9/ipc/shm.c	2006-04-20 11:59:03.000000000 +0100
+++ linux/ipc/shm.c	2006-04-20 16:57:36.000000000 +0100
@@ -161,8 +161,6 @@ static int shm_mmap(struct file * file, 
 	ret = shmem_mmap(file, vma);
 	if (ret == 0) {
 		vma->vm_ops = &shm_vm_ops;
-		if (!(vma->vm_flags & VM_WRITE))
-			vma->vm_flags &= ~VM_MAYWRITE;
 		shm_inc(file->f_dentry->d_inode->i_ino);
 	}
 
@@ -677,6 +675,8 @@ out:
  */
 long do_shmat(int shmid, char __user *shmaddr, int shmflg, ulong *raddr)
 {
+	struct mm_struct *mm = current->mm;
+	struct vm_area_struct *vma;
 	struct shmid_kernel *shp;
 	unsigned long addr;
 	unsigned long size;
@@ -684,7 +684,7 @@ long do_shmat(int shmid, char __user *sh
 	int    err;
 	unsigned long flags;
 	unsigned long prot;
-	unsigned long o_flags;
+	int maywrite;
 	int acc_mode;
 	void *user_addr;
 
@@ -711,11 +711,11 @@ long do_shmat(int shmid, char __user *sh
 
 	if (shmflg & SHM_RDONLY) {
 		prot = PROT_READ;
-		o_flags = O_RDONLY;
+		maywrite = 0;
 		acc_mode = S_IRUGO;
 	} else {
 		prot = PROT_READ | PROT_WRITE;
-		o_flags = O_RDWR;
+		maywrite = 1;
 		acc_mode = S_IRUGO | S_IWUGO;
 	}
 	if (shmflg & SHM_EXEC) {
@@ -748,30 +748,43 @@ long do_shmat(int shmid, char __user *sh
 		shm_unlock(shp);
 		return err;
 	}
-		
+
+	if (!maywrite && !ipcperms_dac(&shp->shm_perm, S_IWUGO))
+		maywrite = 1;
+
 	file = shp->shm_file;
 	size = i_size_read(file->f_dentry->d_inode);
 	shp->shm_nattch++;
 	shm_unlock(shp);
 
-	down_write(&current->mm->mmap_sem);
+	down_write(&mm->mmap_sem);
 	if (addr && !(shmflg & SHM_REMAP)) {
 		user_addr = ERR_PTR(-EINVAL);
-		if (find_vma_intersection(current->mm, addr, addr + size))
+		if (find_vma_intersection(mm, addr, addr + size))
 			goto invalid;
 		/*
 		 * If shm segment goes below stack, make sure there is some
 		 * space left for the stack to grow (at least 4 pages).
 		 */
-		if (addr < current->mm->start_stack &&
-		    addr > current->mm->start_stack - size - PAGE_SIZE * 5)
+		if (addr < mm->start_stack &&
+		    addr > mm->start_stack - size - PAGE_SIZE * 5)
 			goto invalid;
 	}
-		
+
 	user_addr = (void*) do_mmap (file, addr, size, prot, flags, 0);
 
+	if (!maywrite && !IS_ERR(user_addr)) {
+		/*
+		 * Prevent mprotect from giving write permission later on.
+		 * We would prefer just to clear VM_MAYWRITE from a readonly
+		 * attachment in shm_mmap, but it seems that JVM has got into
+		 * the habit of attaching readonly then mprotecting to write.
+		 */
+		vma = find_vma(mm, (unsigned long) user_addr);
+		vma->vm_flags &= ~VM_MAYWRITE;
+	}
 invalid:
-	up_write(&current->mm->mmap_sem);
+	up_write(&mm->mmap_sem);
 
 	down (&shm_ids.sem);
 	if(!(shp = shm_lock(shmid)))
--- 2.6.16.9/ipc/util.c	2006-03-20 05:53:29.000000000 +0000
+++ linux/ipc/util.c	2006-04-20 16:57:36.000000000 +0100
@@ -464,7 +464,7 @@ void ipc_rcu_putref(void *ptr)
  *	to ipc resources. return 0 if allowed
  */
  
-int ipcperms (struct kern_ipc_perm *ipcp, short flag)
+int ipcperms_dac(struct kern_ipc_perm *ipcp, short flag)
 {	/* flag will most probably be 0 or S_...UGO from <linux/stat.h> */
 	int requested_mode, granted_mode;
 
@@ -478,7 +478,13 @@ int ipcperms (struct kern_ipc_perm *ipcp
 	if ((requested_mode & ~granted_mode & 0007) && 
 	    !capable(CAP_IPC_OWNER))
 		return -1;
+	return 0;
+}
 
+int ipcperms(struct kern_ipc_perm *ipcp, short flag)
+{
+	if (ipcperms_dac(ipcp, flag))
+		return -1;
 	return security_ipc_permission(ipcp, flag);
 }
 
--- 2.6.16.9/ipc/util.h	2006-03-20 05:53:29.000000000 +0000
+++ linux/ipc/util.h	2006-04-20 16:57:36.000000000 +0100
@@ -47,7 +47,8 @@ int ipc_addid(struct ipc_ids* ids, struc
 /* must be called with both locks acquired. */
 struct kern_ipc_perm* ipc_rmid(struct ipc_ids* ids, int id);
 
-int ipcperms (struct kern_ipc_perm *ipcp, short flg);
+int ipcperms_dac(struct kern_ipc_perm *ipcp, short flag);
+int ipcperms(struct kern_ipc_perm *ipcp, short flag);
 
 /* for rare, potentially huge allocations.
  * both function can sleep

Re: [stable] 2.6.16.6 breaks java... sort of

[David Wilk]

Yes, absolutely.  I'd love to help test.

I feel obligated to point out that this problem only occured when I
had the Xmx set to 768MB on a machine with only 1GB of ram (3GB total
VM), although it might have caused problems when the JVM eventually
requested more heap space (without the initial grab that Xmx=768m
forces).

I applaud all efforts to secure the kernel, and wouldn't want to
stifle that effort in any way.

I'll give this a whirl and get right back to you.

thanks,
Dave

On 4/20/06, Hugh Dickins <hugh@veritas.com> wrote:
> On Wed, 19 Apr 2006, Hugh Dickins wrote:
> > On Wed, 19 Apr 2006, Greg KH wrote:
> > > On Wed, Apr 19, 2006 at 12:52:33PM -0600, David Wilk wrote:
> > > >
> > > > I think an issue was introduced with mprotect (the first patch in
> > > > 2.6.16.6).  With 2.6.16.5, tomcat runs fine (in sun-1.5), but in
> > > > 2.6.16.7, the JVM bails out complaining that it couldn't allocate
> > > > enough heap space.
> >
> > Neither expected nor satisfactory.  Sorry about that.  We were hoping
> > the straightforward shm/mprotect fix would be good enough, but it
> > appears not.  JVM is probably doing something we can allow with a
> > more complicated patch, but it _might_ turn out to be doing something
> > we simply cannot allow: I'll hope for the first and work out a patch
> > for that; but won't be ready to post it until tomorrow.
>
> David, would you please try this patch on top of your 2.6.16.7 or later.
> The first hunk undoes the problematic patch, the remainder does it in a
> more permissive way.  Aesthetically, not as satisfactory as the previous
> patch; but it's important that we not break userspace, unless security
> absolutely demands.  Please let us know how this fares: thanks.
>
> Hugh
>
> --- 2.6.16.9/ipc/shm.c  2006-04-20 11:59:03.000000000 +0100
> +++ linux/ipc/shm.c     2006-04-20 16:57:36.000000000 +0100
> @@ -161,8 +161,6 @@ static int shm_mmap(struct file * file,
>         ret = shmem_mmap(file, vma);
>         if (ret == 0) {
>                 vma->vm_ops = &shm_vm_ops;
> -               if (!(vma->vm_flags & VM_WRITE))
> -                       vma->vm_flags &= ~VM_MAYWRITE;
>                 shm_inc(file->f_dentry->d_inode->i_ino);
>         }
>
> @@ -677,6 +675,8 @@ out:
>   */
>  long do_shmat(int shmid, char __user *shmaddr, int shmflg, ulong *raddr)
>  {
> +       struct mm_struct *mm = current->mm;
> +       struct vm_area_struct *vma;
>         struct shmid_kernel *shp;
>         unsigned long addr;
>         unsigned long size;
> @@ -684,7 +684,7 @@ long do_shmat(int shmid, char __user *sh
>         int    err;
>         unsigned long flags;
>         unsigned long prot;
> -       unsigned long o_flags;
> +       int maywrite;
>         int acc_mode;
>         void *user_addr;
>
> @@ -711,11 +711,11 @@ long do_shmat(int shmid, char __user *sh
>
>         if (shmflg & SHM_RDONLY) {
>                 prot = PROT_READ;
> -               o_flags = O_RDONLY;
> +               maywrite = 0;
>                 acc_mode = S_IRUGO;
>         } else {
>                 prot = PROT_READ | PROT_WRITE;
> -               o_flags = O_RDWR;
> +               maywrite = 1;
>                 acc_mode = S_IRUGO | S_IWUGO;
>         }
>         if (shmflg & SHM_EXEC) {
> @@ -748,30 +748,43 @@ long do_shmat(int shmid, char __user *sh
>                 shm_unlock(shp);
>                 return err;
>         }
> -
> +
> +       if (!maywrite && !ipcperms_dac(&shp->shm_perm, S_IWUGO))
> +               maywrite = 1;
> +
>         file = shp->shm_file;
>         size = i_size_read(file->f_dentry->d_inode);
>         shp->shm_nattch++;
>         shm_unlock(shp);
>
> -       down_write(&current->mm->mmap_sem);
> +       down_write(&mm->mmap_sem);
>         if (addr && !(shmflg & SHM_REMAP)) {
>                 user_addr = ERR_PTR(-EINVAL);
> -               if (find_vma_intersection(current->mm, addr, addr + size))
> +               if (find_vma_intersection(mm, addr, addr + size))
>                         goto invalid;
>                 /*
>                  * If shm segment goes below stack, make sure there is some
>                  * space left for the stack to grow (at least 4 pages).
>                  */
> -               if (addr < current->mm->start_stack &&
> -                   addr > current->mm->start_stack - size - PAGE_SIZE * 5)
> +               if (addr < mm->start_stack &&
> +                   addr > mm->start_stack - size - PAGE_SIZE * 5)
>                         goto invalid;
>         }
> -
> +
>         user_addr = (void*) do_mmap (file, addr, size, prot, flags, 0);
>
> +       if (!maywrite && !IS_ERR(user_addr)) {
> +               /*
> +                * Prevent mprotect from giving write permission later on.
> +                * We would prefer just to clear VM_MAYWRITE from a readonly
> +                * attachment in shm_mmap, but it seems that JVM has got into
> +                * the habit of attaching readonly then mprotecting to write.
> +                */
> +               vma = find_vma(mm, (unsigned long) user_addr);
> +               vma->vm_flags &= ~VM_MAYWRITE;
> +       }
>  invalid:
> -       up_write(&current->mm->mmap_sem);
> +       up_write(&mm->mmap_sem);
>
>         down (&shm_ids.sem);
>         if(!(shp = shm_lock(shmid)))
> --- 2.6.16.9/ipc/util.c 2006-03-20 05:53:29.000000000 +0000
> +++ linux/ipc/util.c    2006-04-20 16:57:36.000000000 +0100
> @@ -464,7 +464,7 @@ void ipc_rcu_putref(void *ptr)
>   *     to ipc resources. return 0 if allowed
>   */
>
> -int ipcperms (struct kern_ipc_perm *ipcp, short flag)
> +int ipcperms_dac(struct kern_ipc_perm *ipcp, short flag)
>  {      /* flag will most probably be 0 or S_...UGO from <linux/stat.h> */
>         int requested_mode, granted_mode;
>
> @@ -478,7 +478,13 @@ int ipcperms (struct kern_ipc_perm *ipcp
>         if ((requested_mode & ~granted_mode & 0007) &&
>             !capable(CAP_IPC_OWNER))
>                 return -1;
> +       return 0;
> +}
>
> +int ipcperms(struct kern_ipc_perm *ipcp, short flag)
> +{
> +       if (ipcperms_dac(ipcp, flag))
> +               return -1;
>         return security_ipc_permission(ipcp, flag);
>  }
>
> --- 2.6.16.9/ipc/util.h 2006-03-20 05:53:29.000000000 +0000
> +++ linux/ipc/util.h    2006-04-20 16:57:36.000000000 +0100
> @@ -47,7 +47,8 @@ int ipc_addid(struct ipc_ids* ids, struc
>  /* must be called with both locks acquired. */
>  struct kern_ipc_perm* ipc_rmid(struct ipc_ids* ids, int id);
>
> -int ipcperms (struct kern_ipc_perm *ipcp, short flg);
> +int ipcperms_dac(struct kern_ipc_perm *ipcp, short flag);
> +int ipcperms(struct kern_ipc_perm *ipcp, short flag);
>
>  /* for rare, potentially huge allocations.
>   * both function can sleep
>

Re: [stable] 2.6.16.6 breaks java... sort of

[David Wilk]

Ok, on my first test system (lowest amount of ram) a 2.6.16.9 kernel
patched with the patch you provided works fine.  I'll throw it on some
other systems and we'll see how it does.

On 4/20/06, Hugh Dickins <hugh@veritas.com> wrote:
> On Wed, 19 Apr 2006, Hugh Dickins wrote:
> > On Wed, 19 Apr 2006, Greg KH wrote:
> > > On Wed, Apr 19, 2006 at 12:52:33PM -0600, David Wilk wrote:
> > > >
> > > > I think an issue was introduced with mprotect (the first patch in
> > > > 2.6.16.6).  With 2.6.16.5, tomcat runs fine (in sun-1.5), but in
> > > > 2.6.16.7, the JVM bails out complaining that it couldn't allocate
> > > > enough heap space.
> >
> > Neither expected nor satisfactory.  Sorry about that.  We were hoping
> > the straightforward shm/mprotect fix would be good enough, but it
> > appears not.  JVM is probably doing something we can allow with a
> > more complicated patch, but it _might_ turn out to be doing something
> > we simply cannot allow: I'll hope for the first and work out a patch
> > for that; but won't be ready to post it until tomorrow.
>
> David, would you please try this patch on top of your 2.6.16.7 or later.
> The first hunk undoes the problematic patch, the remainder does it in a
> more permissive way.  Aesthetically, not as satisfactory as the previous
> patch; but it's important that we not break userspace, unless security
> absolutely demands.  Please let us know how this fares: thanks.
>
> Hugh
>
> --- 2.6.16.9/ipc/shm.c  2006-04-20 11:59:03.000000000 +0100
> +++ linux/ipc/shm.c     2006-04-20 16:57:36.000000000 +0100
> @@ -161,8 +161,6 @@ static int shm_mmap(struct file * file,
>         ret = shmem_mmap(file, vma);
>         if (ret == 0) {
>                 vma->vm_ops = &shm_vm_ops;
> -               if (!(vma->vm_flags & VM_WRITE))
> -                       vma->vm_flags &= ~VM_MAYWRITE;
>                 shm_inc(file->f_dentry->d_inode->i_ino);
>         }
>
> @@ -677,6 +675,8 @@ out:
>   */
>  long do_shmat(int shmid, char __user *shmaddr, int shmflg, ulong *raddr)
>  {
> +       struct mm_struct *mm = current->mm;
> +       struct vm_area_struct *vma;
>         struct shmid_kernel *shp;
>         unsigned long addr;
>         unsigned long size;
> @@ -684,7 +684,7 @@ long do_shmat(int shmid, char __user *sh
>         int    err;
>         unsigned long flags;
>         unsigned long prot;
> -       unsigned long o_flags;
> +       int maywrite;
>         int acc_mode;
>         void *user_addr;
>
> @@ -711,11 +711,11 @@ long do_shmat(int shmid, char __user *sh
>
>         if (shmflg & SHM_RDONLY) {
>                 prot = PROT_READ;
> -               o_flags = O_RDONLY;
> +               maywrite = 0;
>                 acc_mode = S_IRUGO;
>         } else {
>                 prot = PROT_READ | PROT_WRITE;
> -               o_flags = O_RDWR;
> +               maywrite = 1;
>                 acc_mode = S_IRUGO | S_IWUGO;
>         }
>         if (shmflg & SHM_EXEC) {
> @@ -748,30 +748,43 @@ long do_shmat(int shmid, char __user *sh
>                 shm_unlock(shp);
>                 return err;
>         }
> -
> +
> +       if (!maywrite && !ipcperms_dac(&shp->shm_perm, S_IWUGO))
> +               maywrite = 1;
> +
>         file = shp->shm_file;
>         size = i_size_read(file->f_dentry->d_inode);
>         shp->shm_nattch++;
>         shm_unlock(shp);
>
> -       down_write(&current->mm->mmap_sem);
> +       down_write(&mm->mmap_sem);
>         if (addr && !(shmflg & SHM_REMAP)) {
>                 user_addr = ERR_PTR(-EINVAL);
> -               if (find_vma_intersection(current->mm, addr, addr + size))
> +               if (find_vma_intersection(mm, addr, addr + size))
>                         goto invalid;
>                 /*
>                  * If shm segment goes below stack, make sure there is some
>                  * space left for the stack to grow (at least 4 pages).
>                  */
> -               if (addr < current->mm->start_stack &&
> -                   addr > current->mm->start_stack - size - PAGE_SIZE * 5)
> +               if (addr < mm->start_stack &&
> +                   addr > mm->start_stack - size - PAGE_SIZE * 5)
>                         goto invalid;
>         }
> -
> +
>         user_addr = (void*) do_mmap (file, addr, size, prot, flags, 0);
>
> +       if (!maywrite && !IS_ERR(user_addr)) {
> +               /*
> +                * Prevent mprotect from giving write permission later on.
> +                * We would prefer just to clear VM_MAYWRITE from a readonly
> +                * attachment in shm_mmap, but it seems that JVM has got into
> +                * the habit of attaching readonly then mprotecting to write.
> +                */
> +               vma = find_vma(mm, (unsigned long) user_addr);
> +               vma->vm_flags &= ~VM_MAYWRITE;
> +       }
>  invalid:
> -       up_write(&current->mm->mmap_sem);
> +       up_write(&mm->mmap_sem);
>
>         down (&shm_ids.sem);
>         if(!(shp = shm_lock(shmid)))
> --- 2.6.16.9/ipc/util.c 2006-03-20 05:53:29.000000000 +0000
> +++ linux/ipc/util.c    2006-04-20 16:57:36.000000000 +0100
> @@ -464,7 +464,7 @@ void ipc_rcu_putref(void *ptr)
>   *     to ipc resources. return 0 if allowed
>   */
>
> -int ipcperms (struct kern_ipc_perm *ipcp, short flag)
> +int ipcperms_dac(struct kern_ipc_perm *ipcp, short flag)
>  {      /* flag will most probably be 0 or S_...UGO from <linux/stat.h> */
>         int requested_mode, granted_mode;
>
> @@ -478,7 +478,13 @@ int ipcperms (struct kern_ipc_perm *ipcp
>         if ((requested_mode & ~granted_mode & 0007) &&
>             !capable(CAP_IPC_OWNER))
>                 return -1;
> +       return 0;
> +}
>
> +int ipcperms(struct kern_ipc_perm *ipcp, short flag)
> +{
> +       if (ipcperms_dac(ipcp, flag))
> +               return -1;
>         return security_ipc_permission(ipcp, flag);
>  }
>
> --- 2.6.16.9/ipc/util.h 2006-03-20 05:53:29.000000000 +0000
> +++ linux/ipc/util.h    2006-04-20 16:57:36.000000000 +0100
> @@ -47,7 +47,8 @@ int ipc_addid(struct ipc_ids* ids, struc
>  /* must be called with both locks acquired. */
>  struct kern_ipc_perm* ipc_rmid(struct ipc_ids* ids, int id);
>
> -int ipcperms (struct kern_ipc_perm *ipcp, short flg);
> +int ipcperms_dac(struct kern_ipc_perm *ipcp, short flag);
> +int ipcperms(struct kern_ipc_perm *ipcp, short flag);
>
>  /* for rare, potentially huge allocations.
>   * both function can sleep
>

Re: [stable] 2.6.16.6 breaks java... sort of

[Chris Wright]

* David Wilk (davidwilk@gmail.com) wrote:
> Ok, on my first test system (lowest amount of ram) a 2.6.16.9 kernel
> patched with the patch you provided works fine.  I'll throw it on some
> other systems and we'll see how it does.

Was that same (lowest amount of ram) system failing before Hugh's patch
with vanilla 2.6.16.9?  What we're looking for is to see if the app
was doing:

1) shmget(0444) [RDONLY], shmat(SHM_RDONLY), mprotect(PROT_WRITE)
or
2) shmget(0666) [RDWR], shmat(SHM_RDONLY), mprotect(PROT_WRITE)

The first is definitely a bug, and that's what the original patch
was closing.  The second is technically (as in POSIX) undefined, and
the original patch treated it as a bug as well.  Hugh's additional
patch allows this behaviour, as it's vaguely similar to open(RDWR),
mmap(PROT_READ), mprotect(PROT_WRITE), which is legitimate.  So we're
trying to determine if that's what your app is doing.  strace output
may be unwieldy, but that (or using syscall audit) would be helpful
to clarify.

thanks,
-chris

Re: [stable] 2.6.16.6 breaks java... sort of

[David Wilk]

right, sorry, I forgot to try strace.  let me see what I can do.

On 4/21/06, Chris Wright <chrisw@sous-sol.org> wrote:
> * David Wilk (davidwilk@gmail.com) wrote:
> > Ok, on my first test system (lowest amount of ram) a 2.6.16.9 kernel
> > patched with the patch you provided works fine.  I'll throw it on some
> > other systems and we'll see how it does.
>
> Was that same (lowest amount of ram) system failing before Hugh's patch
> with vanilla 2.6.16.9?  What we're looking for is to see if the app
> was doing:
>
> 1) shmget(0444) [RDONLY], shmat(SHM_RDONLY), mprotect(PROT_WRITE)
> or
> 2) shmget(0666) [RDWR], shmat(SHM_RDONLY), mprotect(PROT_WRITE)
>
> The first is definitely a bug, and that's what the original patch
> was closing.  The second is technically (as in POSIX) undefined, and
> the original patch treated it as a bug as well.  Hugh's additional
> patch allows this behaviour, as it's vaguely similar to open(RDWR),
> mmap(PROT_READ), mprotect(PROT_WRITE), which is legitimate.  So we're
> trying to determine if that's what your app is doing.  strace output
> may be unwieldy, but that (or using syscall audit) would be helpful
> to clarify.
>
> thanks,
> -chris
>

Re: [stable] 2.6.16.6 breaks java... sort of

[David Wilk]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

I finally got strace into the tomcat startup scripts properly and
grabbed the attached output.  I don't see any of the two lines you
propose.  I hope you guys can find this useful.

On 4/21/06, Chris Wright <chrisw@sous-sol.org> wrote:
> * David Wilk (davidwilk@gmail.com) wrote:
> > Ok, on my first test system (lowest amount of ram) a 2.6.16.9 kernel
> > patched with the patch you provided works fine.  I'll throw it on some
> > other systems and we'll see how it does.
>
> Was that same (lowest amount of ram) system failing before Hugh's patch
> with vanilla 2.6.16.9?  What we're looking for is to see if the app
> was doing:
>
> 1) shmget(0444) [RDONLY], shmat(SHM_RDONLY), mprotect(PROT_WRITE)
> or
> 2) shmget(0666) [RDWR], shmat(SHM_RDONLY), mprotect(PROT_WRITE)
>
> The first is definitely a bug, and that's what the original patch
> was closing.  The second is technically (as in POSIX) undefined, and
> the original patch treated it as a bug as well.  Hugh's additional
> patch allows this behaviour, as it's vaguely similar to open(RDWR),
> mmap(PROT_READ), mprotect(PROT_WRITE), which is legitimate.  So we're
> trying to determine if that's what your app is doing.  strace output
> may be unwieldy, but that (or using syscall audit) would be helpful
> to clarify.
>
> thanks,
> -chris
>

[-- Attachment #2: tomcat5_strace.txt --]
[-- Type: text/plain, Size: 33108 bytes --]

execve("/usr/lib/jvm/java/bin/java", ["/usr/lib/jvm/java/bin/java", "-Xmx768m", "-Djava.awt.headless=true", "-Dcom.sun.management.jmxremote=true", "-Dcom.sun.management.jmxremote.port=9004", "-Dcom.sun.management.jmxremote.ssl=false", "-Dcom.sun.management.jmxremote.authenticate=false", "-server", "-Djava.endorsed.dirs=/usr/share/tomcat5/common/endorsed", "-classpath", "/usr/lib/jvm/java/lib/tools.jar:/usr/share/tomcat5/bin/bootstrap.jar:/usr/share/tomcat5/bin/commons-logging-api.jar", "-Dcatalina.base=/usr/share/tomcat5", "-Dcatalina.home=/usr/share/tomcat5", "-Djava.io.tmpdir=/usr/share/tomcat5/temp", "org.apache.catalina.startup.Bootstrap", "start"], [/* 21 vars */]) = 0
uname({sys="Linux", node="davetest2", ...}) = 0
brk(0)                                  = 0x8059000
open("/etc/ld.so.preload", O_RDONLY)    = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=18568, ...}) = 0
old_mmap(NULL, 18568, PROT_READ, MAP_PRIVATE, 3, 0) = 0x37f29000
close(3)                                = 0
open("/lib/tls/libpthread.so.0", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0G\0\000"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=98004, ...}) = 0
old_mmap(NULL, 65140, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x37f19000
old_mmap(0x37f26000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xc000) = 0x37f26000
old_mmap(0x37f27000, 7796, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x37f27000
close(3)                                = 0
open("/lib/libdl.so.2", O_RDONLY)       = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260\32"..., 512) = 512
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f18000
fstat64(3, {st_mode=S_IFREG|0755, st_size=14868, ...}) = 0
old_mmap(NULL, 12244, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x37f15000
old_mmap(0x37f17000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x1000) = 0x37f17000
close(3)                                = 0
open("/lib/tls/libc.so.6", O_RDONLY)    = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200X\1"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1573120, ...}) = 0
old_mmap(NULL, 1280524, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x37ddc000
old_mmap(0x37f0f000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x132000) = 0x37f0f000
old_mmap(0x37f12000, 10764, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x37f12000
close(3)                                = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37ddb000
set_thread_area({entry_number:-1 -> 6, base_addr:0x37ddb080, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
munmap(0x37f29000, 18568)               = 0
bind(937275592, {sin_family=0xdb98 /* AF_??? */, {sa_family=56216, sa_data="\0\0\230=\3647 \243\3637Z\317\3617"}, 938634160) = 12273
rt_sigaction(SIGRTMIN, {0x37f1d660, [], SA_RESTORER|SA_SIGINFO, 0x37f23f80}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN], NULL, 8) = 0
getrlimit(0x3, 0x3fe3f9b0)              = 0
brk(0)                                  = 0x8059000
brk(0x807a000)                          = 0x807a000
brk(0)                                  = 0x807a000
readlink("/proc/self/exe", "/usr/java/jdk1.5.0_06/bin/java", 4095) = 30
access("/usr/java/jdk1.5.0_06/lib/i386/libjava.so", F_OK) = -1 ENOENT (No such file or directory)
access("/usr/java/jdk1.5.0_06/jre/lib/i386/libjava.so", F_OK) = 0
open("/usr/java/jdk1.5.0_06/jre/lib/i386/jvm.cfg", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0444, st_size=689, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f2d000
read(3, "#\n# @(#)jvm.cfg\t1.8 04/02/02\n# \n"..., 4096) = 689
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x37f2d000, 4096)                = 0
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/server/libjvm.so", {st_mode=S_IFREG|0755, st_size=7330509, ...}) = 0
getgid32()                              = 91
getegid32()                             = 91
getuid32()                              = 91
geteuid32()                             = 91
execve("/usr/java/jdk1.5.0_06/bin/java", ["/usr/lib/jvm/java/bin/java", "-Xmx768m", "-Djava.awt.headless=true", "-Dcom.sun.management.jmxremote=t"..., "-Dcom.sun.management.jmxremote.p"..., "-Dcom.sun.management.jmxremote.s"..., "-Dcom.sun.management.jmxremote.a"..., "-server", "-Djava.endorsed.dirs=/usr/share/"..., "-classpath", "/usr/lib/jvm/java/lib/tools.jar:"..., "-Dcatalina.base=/usr/share/tomca"..., "-Dcatalina.home=/usr/share/tomca"..., "-Djava.io.tmpdir=/usr/share/tomc"..., "org.apache.catalina.startup.Boot"..., "start", ...], [/* 22 vars */]) = 0
uname({sys="Linux", node="davetest2", ...}) = 0
brk(0)                                  = 0x8059000
open("/etc/ld.so.preload", O_RDONLY)    = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/tls/i686/mmx/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/server/tls/i686/mmx", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/tls/i686/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/server/tls/i686", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/tls/mmx/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/server/tls/mmx", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/tls/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/server/tls", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/i686/mmx/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/server/i686/mmx", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/i686/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/server/i686", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/mmx/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/server/mmx", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/server", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/usr/java/jdk1.5.0_06/jre/lib/i386/tls/i686/mmx/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/tls/i686/mmx", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/tls/i686/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/tls/i686", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/tls/mmx/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/tls/mmx", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/tls/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/tls", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/i686/mmx/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/i686/mmx", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/i686/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/i686", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/mmx/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/mmx", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/usr/java/jdk1.5.0_06/jre/../lib/i386/tls/i686/mmx/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/../lib/i386/tls/i686/mmx", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/../lib/i386/tls/i686/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/../lib/i386/tls/i686", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/../lib/i386/tls/mmx/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/../lib/i386/tls/mmx", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/../lib/i386/tls/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/../lib/i386/tls", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/../lib/i386/i686/mmx/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/../lib/i386/i686/mmx", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/../lib/i386/i686/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/../lib/i386/i686", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/../lib/i386/mmx/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/../lib/i386/mmx", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/../lib/i386/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/../lib/i386", 0x3fd513d0) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=18568, ...}) = 0
old_mmap(NULL, 18568, PROT_READ, MAP_PRIVATE, 3, 0) = 0x37f3b000
close(3)                                = 0
open("/lib/tls/libpthread.so.0", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0G\0\000"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=98004, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f3a000
old_mmap(NULL, 65140, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x37f2a000
old_mmap(0x37f37000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xc000) = 0x37f37000
old_mmap(0x37f38000, 7796, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x37f38000
close(3)                                = 0
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib/libdl.so.2", O_RDONLY)       = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260\32"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=14868, ...}) = 0
old_mmap(NULL, 12244, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x37f27000
old_mmap(0x37f29000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x1000) = 0x37f29000
close(3)                                = 0
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib/tls/libc.so.6", O_RDONLY)    = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200X\1"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1573120, ...}) = 0
old_mmap(NULL, 1280524, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x37dee000
old_mmap(0x37f21000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x132000) = 0x37f21000
old_mmap(0x37f24000, 10764, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x37f24000
close(3)                                = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37ded000
set_thread_area({entry_number:-1 -> 6, base_addr:0x37ded080, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
munmap(0x37f3b000, 18568)               = 0
bind(937349320, {sin_family=0xdb98 /* AF_??? */, {sa_family=56216, sa_data="\0\0\0\240\3637 \303\3647Z\337\3627"}, 938703792) = 12273
rt_sigaction(SIGRTMIN, {0x37f2e660, [], SA_RESTORER|SA_SIGINFO, 0x37f34f80}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN], NULL, 8) = 0
getrlimit(0x3, 0x3fd51be0)              = 0
brk(0)                                  = 0x8059000
brk(0x807a000)                          = 0x807a000
brk(0)                                  = 0x807a000
readlink("/proc/self/exe", "/usr/java/jdk1.5.0_06/bin/java", 4095) = 30
access("/usr/java/jdk1.5.0_06/lib/i386/libjava.so", F_OK) = -1 ENOENT (No such file or directory)
access("/usr/java/jdk1.5.0_06/jre/lib/i386/libjava.so", F_OK) = 0
open("/usr/java/jdk1.5.0_06/jre/lib/i386/jvm.cfg", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0444, st_size=689, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f3f000
read(3, "#\n# @(#)jvm.cfg\t1.8 04/02/02\n# \n"..., 4096) = 689
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x37f3f000, 4096)                = 0
stat64("/usr/java/jdk1.5.0_06/jre/lib/i386/server/libjvm.so", {st_mode=S_IFREG|0755, st_size=7330509, ...}) = 0
getgid32()                              = 91
getegid32()                             = 91
getuid32()                              = 91
geteuid32()                             = 91
futex(0x37f29fd0, FUTEX_WAKE, 2147483647, NULL) = 0
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/libjvm.so", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\360\326"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=7330509, ...}) = 0
old_mmap(NULL, 10246460, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x37427000
old_mmap(0x37974000, 401408, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x54d000) = 0x37974000
old_mmap(0x379d6000, 4286780, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x379d6000
close(3)                                = 0
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/libm.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/libm.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=18568, ...}) = 0
old_mmap(NULL, 18568, PROT_READ, MAP_PRIVATE, 3, 0) = 0x37f3b000
close(3)                                = 0
open("/lib/tls/libm.so.6", O_RDONLY)    = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\3604\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=213508, ...}) = 0
old_mmap(NULL, 135616, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x37405000
old_mmap(0x37426000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x21000) = 0x37426000
close(3)                                = 0
munmap(0x37f3b000, 18568)               = 0
getpid()                                = 12273
open("/etc/mtab", O_RDONLY)             = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=185, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f3f000
read(3, "/dev/ROOT / ext3 rw,data=journal"..., 4096) = 185
close(3)                                = 0
munmap(0x37f3f000, 4096)                = 0
open("/proc/stat", O_RDONLY)            = 3
fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f3f000
read(3, "cpu  1880 11 10219 286673 87 149"..., 1024) = 632
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x37f3f000, 4096)                = 0
open("/etc/mtab", O_RDONLY)             = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=185, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f3f000
read(3, "/dev/ROOT / ext3 rw,data=journal"..., 4096) = 185
close(3)                                = 0
munmap(0x37f3f000, 4096)                = 0
open("/proc/meminfo", O_RDONLY)         = 3
fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f3f000
read(3, "MemTotal:       254684 kB\nMemFre"..., 1024) = 598
close(3)                                = 0
munmap(0x37f3f000, 4096)                = 0
gettimeofday({1145655542, 541122}, NULL) = 0
lstat64("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre/lib/i386", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre/lib/i386/server", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre/lib/i386/server/libjvm.so", {st_mode=S_IFREG|0755, st_size=7330509, ...}) = 0
open(".hotspotrc", O_RDONLY)            = -1 ENOENT (No such file or directory)
open("/usr/share/tomcat5/common/endorsed", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
getdents64(0x3, 0x805ad7c, 0x1000, 0x805ad60) = 48
getdents64(0x3, 0x805ad7c, 0x1000, 0x805ad60) = 0
close(3)                                = 0
open("/etc/mtab", O_RDONLY)             = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=185, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f3f000
read(3, "/dev/ROOT / ext3 rw,data=journal"..., 4096) = 185
close(3)                                = 0
munmap(0x37f3f000, 4096)                = 0
open("/proc/stat", O_RDONLY)            = 3
fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f3f000
read(3, "cpu  1880 11 10219 286673 87 149"..., 1024) = 632
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x37f3f000, 4096)                = 0
gettimeofday({1145655542, 545259}, NULL) = 0
mmap2(NULL, 4096, PROT_READ, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f3f000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f3e000
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigaction(SIGUSR2, {0x3784bbb0, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x37f34f80}, NULL, 8) = 0
rt_sigaction(SIGHUP, NULL, {SIG_DFL}, 8) = 0
rt_sigaction(SIGINT, NULL, {SIG_IGN}, 8) = 0
rt_sigaction(SIGTERM, NULL, {SIG_DFL}, 8) = 0
rt_sigaction(SIGSEGV, NULL, {SIG_DFL}, 8) = 0
rt_sigaction(SIGSEGV, {0x37849860, ~[RTMIN], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x37f34f80}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGPIPE, NULL, {SIG_DFL}, 8) = 0
rt_sigaction(SIGPIPE, {0x37849860, ~[RTMIN], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x37f34f80}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGBUS, NULL, {SIG_DFL}, 8) = 0
rt_sigaction(SIGBUS, {0x37849860, ~[RTMIN], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x37f34f80}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGILL, NULL, {SIG_DFL}, 8) = 0
rt_sigaction(SIGILL, {0x37849860, ~[RTMIN], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x37f34f80}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGFPE, NULL, {SIG_DFL}, 8) = 0
rt_sigaction(SIGFPE, {0x37849860, ~[RTMIN], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x37f34f80}, {SIG_DFL}, 8) = 0
getrlimit(0x3, 0x3fd4f218)              = 0
open("/proc/self/maps", O_RDONLY)       = 3
fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f3d000
read(3, "08048000-08057000 r-xp 00000000 "..., 1024) = 1024
read(3, "p 0000c000 08:01 81189      /lib"..., 1024) = 424
close(3)                                = 0
munmap(0x37f3d000, 4096)                = 0
getrlimit(0x7, 0x3fd4fa50)              = 0
setrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0
open("/usr/java/jdk1.5.0_06/jre/lib/i386/native_threads/libhpi.so", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P\"\0\000"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=27144, ...}) = 0
old_mmap(NULL, 27072, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x373fe000
old_mmap(0x37404000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x6000) = 0x37404000
close(3)                                = 0
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/libnsl.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/libnsl.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=18568, ...}) = 0
old_mmap(NULL, 18568, PROT_READ, MAP_PRIVATE, 3, 0) = 0x373f9000
close(3)                                = 0
open("/lib/libnsl.so.1", O_RDONLY)      = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0 <\0\000"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=91040, ...}) = 0
old_mmap(NULL, 84864, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x373e4000
old_mmap(0x373f6000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x11000) = 0x373f6000
old_mmap(0x373f7000, 7040, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x373f7000
close(3)                                = 0
munmap(0x373f9000, 18568)               = 0
geteuid32()                             = 91
socket(PF_UNIX, SOCK_STREAM, 0)         = 3
connect(3, {sin_family=AF_UNIX, path="/var/run/.nscd_socket"}, 110) = -1 ENOENT (No such file or directory)
close(3)                                = 0
open("/etc/nsswitch.conf", O_RDONLY)    = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=1686, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f3d000
read(3, "#\n# /etc/nsswitch.conf\n#\n# An ex"..., 4096) = 1686
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x37f3d000, 4096)                = 0
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/libnss_files.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/java/jdk1.5.0_06/jre/lib/i386/libnss_files.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=18568, ...}) = 0
old_mmap(NULL, 18568, PROT_READ, MAP_PRIVATE, 3, 0) = 0x373f9000
close(3)                                = 0
open("/lib/libnss_files.so.2", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\35\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=51952, ...}) = 0
old_mmap(NULL, 46748, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x373d8000
old_mmap(0x373e3000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xa000) = 0x373e3000
close(3)                                = 0
munmap(0x373f9000, 18568)               = 0
open("/etc/passwd", O_RDONLY)           = 3
fcntl64(3, F_GETFD)                     = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=1335, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x37f3d000
read(3, "root:x:0:0:root:/root:/bin/sh\nad"..., 4096) = 1335
close(3)                                = 0
munmap(0x37f3d000, 4096)                = 0
open("/tmp/hsperfdata_tomcat", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
lstat64("/tmp/hsperfdata_tomcat", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getdents64(0x3, 0x805dc64, 0x1000, 0x805dc48) = 48
getdents64(0x3, 0x805dc64, 0x1000, 0x805dc48) = 0
close(3)                                = 0
mkdir("/tmp/hsperfdata_tomcat", 0755)   = -1 EEXIST (File exists)
lstat64("/tmp/hsperfdata_tomcat", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/tmp/hsperfdata_tomcat/12273", O_RDWR|O_CREAT|O_TRUNC, 0600) = 3
ftruncate(3, 32768)                     = 0
mmap2(NULL, 32768, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x373d0000
close(3)                                = 0
gettid()                                = 12273
rt_sigprocmask(SIG_SETMASK, ~[RTMIN], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_UNBLOCK, [HUP ILL BUS FPE SEGV USR2 TERM], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [QUIT], NULL, 8) = 0
mmap2(0x3fb55000, 12288, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3fb55000
mprotect(0x3fb55000, 12288, PROT_NONE)  = 0
gettimeofday({1145655542, 560996}, NULL) = 0
gettimeofday({1145655542, 561108}, NULL) = 0
gettimeofday({1145655542, 561312}, NULL) = 0
gettimeofday({1145655542, 561429}, NULL) = 0
gettimeofday({1145655542, 561528}, NULL) = 0
gettimeofday({1145655542, 561791}, NULL) = 0
gettimeofday({1145655542, 561891}, NULL) = 0
gettimeofday({1145655542, 562000}, NULL) = 0
gettimeofday({1145655542, 562098}, NULL) = 0
gettimeofday({1145655542, 562196}, NULL) = 0
gettimeofday({1145655542, 562292}, NULL) = 0
gettimeofday({1145655542, 562407}, NULL) = 0
gettimeofday({1145655542, 562553}, NULL) = 0
gettimeofday({1145655542, 562651}, NULL) = 0
gettimeofday({1145655542, 564154}, NULL) = 0
gettimeofday({1145655542, 564259}, NULL) = 0
gettimeofday({1145655542, 564359}, NULL) = 0
gettimeofday({1145655542, 564456}, NULL) = 0
gettimeofday({1145655542, 564554}, NULL) = 0
gettimeofday({1145655542, 564651}, NULL) = 0
gettimeofday({1145655542, 564903}, NULL) = 0
gettimeofday({1145655542, 565005}, NULL) = 0
gettimeofday({1145655542, 565103}, NULL) = 0
gettimeofday({1145655542, 568194}, NULL) = 0
open("/usr/java/jdk1.5.0_06/jre/lib/i386/libverify.so", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320 \0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=48520, ...}) = 0
old_mmap(NULL, 47428, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x373c4000
old_mmap(0x373cf000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xb000) = 0x373cf000
close(3)                                = 0
open("/usr/java/jdk1.5.0_06/jre/lib/i386/libjava.so", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220\221"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=136824, ...}) = 0
old_mmap(NULL, 139888, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x373a1000
old_mmap(0x373c2000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x20000) = 0x373c2000
close(3)                                = 0
open("/usr/java/jdk1.5.0_06/jre/lib/i386/libzip.so", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260\34"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=82120, ...}) = 0
old_mmap(NULL, 85504, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x3738c000
old_mmap(0x3739f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x12000) = 0x3739f000
close(3)                                = 0
stat64("/usr/java/jdk1.5.0_06/jre/lib/rt.jar", {st_mode=S_IFREG|0644, st_size=39744921, ...}) = 0
lstat64("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre/lib/rt.jar", {st_mode=S_IFREG|0644, st_size=39744921, ...}) = 0
open("/usr/java/jdk1.5.0_06/jre/lib/rt.jar", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=39744921, ...}) = 0
_llseek(3, 0, [39744921], SEEK_END)     = 0
mmap2(NULL, 39744921, PROT_READ, MAP_SHARED, 3, 0) = 0x34da4000
close(3)                                = 0
mmap2(NULL, 430080, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x34d3b000
stat64("/usr/java/jdk1.5.0_06/jre/lib/i18n.jar", 0x3fd4f0dc) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/sunrsasign.jar", 0x3fd4f0dc) = -1 ENOENT (No such file or directory)
stat64("/usr/java/jdk1.5.0_06/jre/lib/jsse.jar", {st_mode=S_IFREG|0644, st_size=542082, ...}) = 0
lstat64("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre/lib/jsse.jar", {st_mode=S_IFREG|0644, st_size=542082, ...}) = 0
open("/usr/java/jdk1.5.0_06/jre/lib/jsse.jar", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=542082, ...}) = 0
_llseek(3, 0, [542082], SEEK_END)       = 0
mmap2(NULL, 542082, PROT_READ, MAP_SHARED, 3, 0) = 0x34cb6000
close(3)                                = 0
stat64("/usr/java/jdk1.5.0_06/jre/lib/jce.jar", {st_mode=S_IFREG|0444, st_size=81799, ...}) = 0
lstat64("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre/lib/jce.jar", {st_mode=S_IFREG|0444, st_size=81799, ...}) = 0
open("/usr/java/jdk1.5.0_06/jre/lib/jce.jar", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0444, st_size=81799, ...}) = 0
_llseek(3, 0, [81799], SEEK_END)        = 0
mmap2(NULL, 81799, PROT_READ, MAP_SHARED, 3, 0) = 0x34ca2000
close(3)                                = 0
stat64("/usr/java/jdk1.5.0_06/jre/lib/charsets.jar", {st_mode=S_IFREG|0644, st_size=8627836, ...}) = 0
lstat64("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/java/jdk1.5.0_06/jre/lib/charsets.jar", {st_mode=S_IFREG|0644, st_size=8627836, ...}) = 0
open("/usr/java/jdk1.5.0_06/jre/lib/charsets.jar", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=8627836, ...}) = 0
_llseek(3, 0, [8627836], SEEK_END)      = 0
mmap2(NULL, 8627836, PROT_READ, MAP_SHARED, 3, 0) = 0x34467000
close(3)                                = 0
stat64("/usr/java/jdk1.5.0_06/jre/classes", 0x3fd4f0dc) = -1 ENOENT (No such file or directory)
mmap2(NULL, 33554432, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x32467000
mmap2(0x32467000, 163840, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x32467000
mmap2(NULL, 524288, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x323e7000
mmap2(0x323e7000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x323e7000
uname({sys="Linux", node="davetest2", ...}) = 0
open("/usr/java/jdk1.5.0_06/jre/lib/i386/server/classes.jsa", O_RDONLY) = -1 ENOENT (No such file or directory)
gettimeofday({1145655542, 705527}, NULL) = 0
gettimeofday({1145655542, 705648}, NULL) = 0
gettimeofday({1145655542, 705748}, NULL) = 0
gettimeofday({1145655542, 706682}, NULL) = 0
gettimeofday({1145655542, 706784}, NULL) = 0
gettimeofday({1145655542, 706894}, NULL) = 0
gettimeofday({1145655542, 706995}, NULL) = 0
gettimeofday({1145655542, 707093}, NULL) = 0
mmap2(NULL, 872415232, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
write(1, "Error occurred during initializa"..., 43) = 43
write(1, "Could not reserve enough space f"..., 46) = 46
write(1, "\n", 1)                       = 1
unlink("/tmp/hsperfdata_tomcat/12273")  = 0
write(2, "Could not create the Java virtua"..., 43) = 43
SYS_252(0x1, 0x1, 0, 0x37f23bd4, 0x37f23bd4

Re: [stable] 2.6.16.6 breaks java... sort of

[Hugh Dickins]

On Fri, 21 Apr 2006, David Wilk wrote:
> I finally got strace into the tomcat startup scripts properly and
> grabbed the attached output.  I don't see any of the two lines you
> propose.  I hope you guys can find this useful.

Thanks for getting that, David.  As you observe, it doesn't involve
shm at all, and the only mprotect is PROT_NONE.  Do the abbreviated
messages in the final lines of the trace fit with the errors you
were originally reporting?  (I think so.)  Or is this particular
trace failing for some other reason, earlier than before, and we
need to try something else to identify the problem?

> mmap2(NULL, 872415232, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
> write(1, "Error occurred during initializa"..., 43) = 43
> write(1, "Could not reserve enough space f"..., 46) = 46
> write(1, "\n", 1)                       = 1
> unlink("/tmp/hsperfdata_tomcat/12273")  = 0
> write(2, "Could not create the Java virtua"..., 43) = 43

To judge by this trace, I'd have to say that your problem has
nothing whatever to do with the shm/mprotect fix in 2.6.16.6,
and we've no evidence yet to complicate that fix.  Interestingly,
nobody else has so far reported any problem with it.

Judging by the mmap addresses throughout the trace (top down, from
0x37f2e000), it looks like you've got CONFIG_VMSPLIT_1G (not a good
choice for a box with only 1G of RAM: whereas CONFIG_VMSPLIT_3G_OPT
would maximize your userspace while avoiding the need for HIGHMEM);
and with the above 832M mmap, the remaining hole in user address
space is just too small to hold it.

But that leaves me quite unable to explain why you should have
thought the shm/mprotect patch responsible, and why you should
find the more complicated version works.  Stack randomization
changes the numbers a little, but I think not enough to explain
how it sometimes could fit 832M in there, sometimes not.

Tell me I'm talking nonsense and we'll have another go:
I guess adding some printks on top of the "replacement"
patch, so it can tell us when it's having an effect.

Hugh

Re: [stable] 2.6.16.6 breaks java... sort of

[David Wilk]

On 4/23/06, Hugh Dickins <hugh@veritas.com> wrote:
> On Fri, 21 Apr 2006, David Wilk wrote:
> > I finally got strace into the tomcat startup scripts properly and
> > grabbed the attached output.  I don't see any of the two lines you
> > propose.  I hope you guys can find this useful.
>
> Thanks for getting that, David.  As you observe, it doesn't involve
> shm at all, and the only mprotect is PROT_NONE.  Do the abbreviated
> messages in the final lines of the trace fit with the errors you
> were originally reporting?  (I think so.)  Or is this particular
> trace failing for some other reason, earlier than before, and we
> need to try something else to identify the problem?

I think this trace was taken while java was doing exactly what it was
doing before.  I actually restarted java many, many times with
2.6.16.9 and watched in amazement as it failed each and every time,
with the exact same error message.  This is tomcat, specifically, and
it would die immediately after startup and it was started the exact
same way with the init script.  So, yeah, I think this trace is
representative.  I have no idea why it doesn't contain what you would
consider relevant.  I'm no programmer, unfortunately.  However, I
would like to help out any way that I can.
>
> > mmap2(NULL, 872415232, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
> > write(1, "Error occurred during initializa"..., 43) = 43
> > write(1, "Could not reserve enough space f"..., 46) = 46
> > write(1, "\n", 1)                       = 1
> > unlink("/tmp/hsperfdata_tomcat/12273")  = 0
> > write(2, "Could not create the Java virtua"..., 43) = 43
>
> To judge by this trace, I'd have to say that your problem has
> nothing whatever to do with the shm/mprotect fix in 2.6.16.6,
> and we've no evidence yet to complicate that fix.  Interestingly,
> nobody else has so far reported any problem with it.

bizaar.  I must say that this patch 'fixing' the problem just cannot
be coincidental.  Tomcat will never start without it, and never fails
with it.
>
> Judging by the mmap addresses throughout the trace (top down, from
> 0x37f2e000), it looks like you've got CONFIG_VMSPLIT_1G (not a good
> choice for a box with only 1G of RAM: whereas CONFIG_VMSPLIT_3G_OPT
> would maximize your userspace while avoiding the need for HIGHMEM);
> and with the above 832M mmap, the remaining hole in user address
> space is just too small to hold it.

well, this is just a test box for a system we deploy on a dual-cpu
server with 4GB of ram.

can you describe the CONFIG_VMSPLIT_3G_OPT and how I might find it in
'make menuconfig'?  is it the second option (3G/1G user/kernel)?  I"ve
got the first option selected which is 3G/1G user/kernel as well, but
different somehow.  As this is new to 2.6.16, I'm not familiar with
the options.  Perhaps my 1GB workstations would benefit from this
second 3G/1G option?
>
> But that leaves me quite unable to explain why you should have
> thought the shm/mprotect patch responsible, and why you should
> find the more complicated version works.  Stack randomization
> changes the numbers a little, but I think not enough to explain
> how it sometimes could fit 832M in there, sometimes not.

Unfortunately I cannot speculate as to the cause, but experimentally
(anecdotally anyway) the patch is 100% effective.
>
> Tell me I'm talking nonsense and we'll have another go:
> I guess adding some printks on top of the "replacement"
> patch, so it can tell us when it's having an effect.

I'd never accuse you of nonsense, but I cannot refute the evidence. ; )

if there is anything else you would like me todo to try to squeeze
more data from this thing, please let me know.
>
> Hugh
>

Re: [stable] 2.6.16.6 breaks java... sort of

[Hugh Dickins]

On Mon, 24 Apr 2006, David Wilk wrote:
> On 4/23/06, Hugh Dickins <hugh@veritas.com> wrote:
> 
> I think this trace was taken while java was doing exactly what it was
> doing before.  I actually restarted java many, many times with
> 2.6.16.9 and watched in amazement as it failed each and every time,
> with the exact same error message.  This is tomcat, specifically, and
> it would die immediately after startup and it was started the exact
> same way with the init script.  So, yeah, I think this trace is
> representative.  I have no idea why it doesn't contain what you would
> consider relevant.  I'm no programmer, unfortunately.  However, I
> would like to help out any way that I can.

Thanks for the clarification.  But I remain utterly perplexed.

I've installed JRE and JDK 1.5 in 1GB box here, I've installed
Tomcat 5.0.28, I've inserted the options from your strace (notably
the -Xmx768m) in my catalina.sh, yet I see no such problem with 2.6.16.9.

> > > mmap2(NULL, 872415232, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
> > > write(1, "Error occurred during initializa"..., 43) = 43
> > > write(1, "Could not reserve enough space f"..., 46) = 46
> > > write(1, "\n", 1)                       = 1
> > > unlink("/tmp/hsperfdata_tomcat/12273")  = 0
> > > write(2, "Could not create the Java virtua"..., 43) = 43
> >
> > To judge by this trace, I'd have to say that your problem has
> > nothing whatever to do with the shm/mprotect fix in 2.6.16.6,
> > and we've no evidence yet to complicate that fix.  Interestingly,
> > nobody else has so far reported any problem with it.
> 
> bizaar.  I must say that this patch 'fixing' the problem just cannot
> be coincidental.  Tomcat will never start without it, and never fails
> with it.

Bizarre indeed.  But at least you've rechecked and banished my doubts.

> > Judging by the mmap addresses throughout the trace (top down, from
> > 0x37f2e000), it looks like you've got CONFIG_VMSPLIT_1G (not a good
> > choice for a box with only 1G of RAM: whereas CONFIG_VMSPLIT_3G_OPT
> > would maximize your userspace while avoiding the need for HIGHMEM);
> > and with the above 832M mmap, the remaining hole in user address
> > space is just too small to hold it.
> 
> well, this is just a test box for a system we deploy on a dual-cpu
> server with 4GB of ram.
> 
> can you describe the CONFIG_VMSPLIT_3G_OPT and how I might find it in
> 'make menuconfig'?  is it the second option (3G/1G user/kernel)?  I"ve

The second one, that's _OPT, yes, which just shaves a little off the
3GB userspace, to avoid the need for HIGHMEM.

> got the first option selected which is 3G/1G user/kernel as well, but
> different somehow.  As this is new to 2.6.16, I'm not familiar with

The first one, this is the traditional layout, the one least likely to
give you any compatibility troubles, so please do stick with it for now.

> the options.  Perhaps my 1GB workstations would benefit from this
> second 3G/1G option?

They would, a little.  Whether it's noticeable, I don't know.
You might prefer to keep a similar configuration for all your
machines.  While we're investigating this particular mystery,
I'd prefer you to stick with what you've got.

> > But that leaves me quite unable to explain why you should have
> > thought the shm/mprotect patch responsible, and why you should
> > find the more complicated version works.  Stack randomization
> > changes the numbers a little, but I think not enough to explain
> > how it sometimes could fit 832M in there, sometimes not.

Clearly I was entirely wrong to accuse you of the wrong VMSPLIT.
Yet I still can't see how either shm or mprotect write is involved.

My latest guess is that whereas my "ulimit -s" shows 8192, yours
will show something much higher, but not "unlimited".   And that
high stack rlimit is pushing the roof of your userspace down, to
the point where the -Xmx868m mapping barely fits.

But though fiddling with my "ulimit -s" does change the layout,
I've not yet been able to reproduce your failure.  And this has
nothing whatever to do with the shm/mprotect issue or patch.

> Unfortunately I cannot speculate as to the cause, but experimentally
> (anecdotally anyway) the patch is 100% effective.
> >
> > Tell me I'm talking nonsense and we'll have another go:
> > I guess adding some printks on top of the "replacement"
> > patch, so it can tell us when it's having an effect.
> 
> I'd never accuse you of nonsense, but I cannot refute the evidence. ; )
> 
> if there is anything else you would like me todo to try to squeeze
> more data from this thing, please let me know.

Well, I guess I'd like to see what "ulimit -a" and "ulimit -H -a"
show on your machine (it's really "ulimit -s" I'm interested in,
but we might as well see the whole lot).  And the output from
"cat /proc/$(pidof java)/maps" while you've got Tomcat running
successfully.  And what is your distro?

But it seems unlikely that any of this information will help me to
make that strange leap from my patches to your experience of them.

Hugh

Re: [stable] 2.6.16.6 breaks java... sort of

[David Wilk]

Ok, I think I need to apologize to everyone here.  I have found the
problem, and it is not with your patch, Hugh.  For some reason, the
config for my 2.6.16.7 source tree had a 1G/3G user/kernel split
configured.  This is very bizaar as I copy my .config from tree to
tree to avoid any changes in the configuration of my test kernels.

I imagine this is the expected behavior when you only have 1G
configured for user space?  right.  I will be sure to include my
/proc/config.gz in the future to prevent this from happening again.

I've tested 2.6.16.7 and 2.6.16.9 and neither of them have the mmap constraint.

again, my apologies.  I thank you for your patience and your hardwork
on the kernel.

thanks,
Dave

On 4/24/06, David Wilk <davidwilk@gmail.com> wrote:
> On 4/23/06, Hugh Dickins <hugh@veritas.com> wrote:
> > On Fri, 21 Apr 2006, David Wilk wrote:
> > > I finally got strace into the tomcat startup scripts properly and
> > > grabbed the attached output.  I don't see any of the two lines you
> > > propose.  I hope you guys can find this useful.
> >
> > Thanks for getting that, David.  As you observe, it doesn't involve
> > shm at all, and the only mprotect is PROT_NONE.  Do the abbreviated
> > messages in the final lines of the trace fit with the errors you
> > were originally reporting?  (I think so.)  Or is this particular
> > trace failing for some other reason, earlier than before, and we
> > need to try something else to identify the problem?
>
> I think this trace was taken while java was doing exactly what it was
> doing before.  I actually restarted java many, many times with
> 2.6.16.9 and watched in amazement as it failed each and every time,
> with the exact same error message.  This is tomcat, specifically, and
> it would die immediately after startup and it was started the exact
> same way with the init script.  So, yeah, I think this trace is
> representative.  I have no idea why it doesn't contain what you would
> consider relevant.  I'm no programmer, unfortunately.  However, I
> would like to help out any way that I can.
> >
> > > mmap2(NULL, 872415232, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
> > > write(1, "Error occurred during initializa"..., 43) = 43
> > > write(1, "Could not reserve enough space f"..., 46) = 46
> > > write(1, "\n", 1)                       = 1
> > > unlink("/tmp/hsperfdata_tomcat/12273")  = 0
> > > write(2, "Could not create the Java virtua"..., 43) = 43
> >
> > To judge by this trace, I'd have to say that your problem has
> > nothing whatever to do with the shm/mprotect fix in 2.6.16.6,
> > and we've no evidence yet to complicate that fix.  Interestingly,
> > nobody else has so far reported any problem with it.
>
> bizaar.  I must say that this patch 'fixing' the problem just cannot
> be coincidental.  Tomcat will never start without it, and never fails
> with it.
> >
> > Judging by the mmap addresses throughout the trace (top down, from
> > 0x37f2e000), it looks like you've got CONFIG_VMSPLIT_1G (not a good
> > choice for a box with only 1G of RAM: whereas CONFIG_VMSPLIT_3G_OPT
> > would maximize your userspace while avoiding the need for HIGHMEM);
> > and with the above 832M mmap, the remaining hole in user address
> > space is just too small to hold it.
>
> well, this is just a test box for a system we deploy on a dual-cpu
> server with 4GB of ram.
>
> can you describe the CONFIG_VMSPLIT_3G_OPT and how I might find it in
> 'make menuconfig'?  is it the second option (3G/1G user/kernel)?  I"ve
> got the first option selected which is 3G/1G user/kernel as well, but
> different somehow.  As this is new to 2.6.16, I'm not familiar with
> the options.  Perhaps my 1GB workstations would benefit from this
> second 3G/1G option?
> >
> > But that leaves me quite unable to explain why you should have
> > thought the shm/mprotect patch responsible, and why you should
> > find the more complicated version works.  Stack randomization
> > changes the numbers a little, but I think not enough to explain
> > how it sometimes could fit 832M in there, sometimes not.
>
> Unfortunately I cannot speculate as to the cause, but experimentally
> (anecdotally anyway) the patch is 100% effective.
> >
> > Tell me I'm talking nonsense and we'll have another go:
> > I guess adding some printks on top of the "replacement"
> > patch, so it can tell us when it's having an effect.
>
> I'd never accuse you of nonsense, but I cannot refute the evidence. ; )
>
> if there is anything else you would like me todo to try to squeeze
> more data from this thing, please let me know.
> >
> > Hugh
> >
>

Re: [stable] 2.6.16.6 breaks java... sort of

[Hugh Dickins]

On Tue, 25 Apr 2006, David Wilk wrote:
> Ok, I think I need to apologize to everyone here.  I have found the
> problem, and it is not with your patch, Hugh.  For some reason, the
> config for my 2.6.16.7 source tree had a 1G/3G user/kernel split
> configured.

Right, that's just the kind of explanation I came up with yesterday.
Phew, I'll relax!  And thanks for the confession, David - absolved.

It remains the case that my patch imposes a new constraint that
could give rise to problems somewhere: but it closes the hole,
and there have been no other reports of problems so far.

Hugh

Re: [stable] 2.6.16.6 breaks java... sort of

[Andi Kleen]

"David Wilk" <davidwilk@gmail.com> writes:

> Ok, I think I need to apologize to everyone here.  I have found the
> problem, and it is not with your patch, Hugh.  For some reason, the
> config for my 2.6.16.7 source tree had a 1G/3G user/kernel split
> configured.  This is very bizaar as I copy my .config from tree to
> tree to avoid any changes in the configuration of my test kernels.

This just shows this dreaded VMSPLIT config was a bad idea in the first
place. There was a reason we didn't have it for such a long time (too
many users get it wrong) and such occurrences just show again that this
is still true.

IMHO it would be best to just remove that option again and require
users who really want to change this to patch their kernels again.

At the very least it should be probably made dependent on CONFIG_EMBEDDED.

-Andi

Mark VMSPLIT EMBEDDED

Too many users get it wrong.

Signed-off-by: Andi Kleen <ak@suse.de>

Index: linux/arch/i386/Kconfig
===================================================================
--- linux.orig/arch/i386/Kconfig
+++ linux/arch/i386/Kconfig
@@ -466,7 +466,7 @@ endchoice
 
 choice
 	depends on EXPERIMENTAL && !X86_PAE
-	prompt "Memory split"
+	prompt "Memory split" if EMBEDDED
 	default VMSPLIT_3G
 	help
 	  Select the desired split between kernel and user memory.

Re: [stable] 2.6.16.6 breaks java... sort of

[Adrian Bunk]

On Wed, Apr 26, 2006 at 07:12:43AM +0200, Andi Kleen wrote:
> "David Wilk" <davidwilk@gmail.com> writes:
> 
> > Ok, I think I need to apologize to everyone here.  I have found the
> > problem, and it is not with your patch, Hugh.  For some reason, the
> > config for my 2.6.16.7 source tree had a 1G/3G user/kernel split
> > configured.  This is very bizaar as I copy my .config from tree to
> > tree to avoid any changes in the configuration of my test kernels.
> 
> This just shows this dreaded VMSPLIT config was a bad idea in the first
> place. There was a reason we didn't have it for such a long time (too
> many users get it wrong) and such occurrences just show again that this
> is still true.
> 
> IMHO it would be best to just remove that option again and require
> users who really want to change this to patch their kernels again.
> 
> At the very least it should be probably made dependent on CONFIG_EMBEDDED.

NAK, EMBEDDED has a different semantics (space savings in memory limited 
situations) that does not apply to this option.

> -Andi
>...

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [stable] 2.6.16.6 breaks java... sort of

[David Wilk]

In defense of having that option available, this was not a case where
the user was deliberately trying to set that option, but rather that
it somehow got set accidentally (still a bit of a mystery as I was not
modifying configs at the time).

On 26 Apr 2006 07:12:43 +0200, Andi Kleen <ak@suse.de> wrote:
> "David Wilk" <davidwilk@gmail.com> writes:
>
> > Ok, I think I need to apologize to everyone here.  I have found the
> > problem, and it is not with your patch, Hugh.  For some reason, the
> > config for my 2.6.16.7 source tree had a 1G/3G user/kernel split
> > configured.  This is very bizaar as I copy my .config from tree to
> > tree to avoid any changes in the configuration of my test kernels.
>
> This just shows this dreaded VMSPLIT config was a bad idea in the first
> place. There was a reason we didn't have it for such a long time (too
> many users get it wrong) and such occurrences just show again that this
> is still true.
>
> IMHO it would be best to just remove that option again and require
> users who really want to change this to patch their kernels again.
>
> At the very least it should be probably made dependent on CONFIG_EMBEDDED.
>
> -Andi
>
> Mark VMSPLIT EMBEDDED
>
> Too many users get it wrong.
>
> Signed-off-by: Andi Kleen <ak@suse.de>
>
> Index: linux/arch/i386/Kconfig
> ===================================================================
> --- linux.orig/arch/i386/Kconfig
> +++ linux/arch/i386/Kconfig
> @@ -466,7 +466,7 @@ endchoice
>
>  choice
>         depends on EXPERIMENTAL && !X86_PAE
> -       prompt "Memory split"
> +       prompt "Memory split" if EMBEDDED
>         default VMSPLIT_3G
>         help
>           Select the desired split between kernel and user memory.
>