MAX_PID changes in 2.5.31

MAX_PID changes in 2.5.31

[Ingo Molnar]

Linus,

afaics, you did the PID_MAX changes in v2.5.31? This is a change i had for
(surprise) threading purposes already, but done a bit differently.

The main problem is that there's the old-style SysV IPC interface that
uses 16-bit PIDs still. All recent SysV applications (linked against glibc
2.2 or newer) use IPC_64, but any application linked against pre-2.2
glibcs will fail. glibc 2.2 was released 2 years ago, is this enough of a
timeout to obsolete the non-IPC_64 interfaces?

if that is the case then can i rip all the non-IPC_64 parts out of ipc/*,
and let non-IPC_64 calls fail? Right now it's silent breakage that
happens.

or, in my threading tree, i introduced a /proc/sys/kernel/pid_max tunable,
which has the safe conservative value of 32K PIDs, but which can be
changed by the admin to have higher PIDs.

[anything more complex than this i think should be ignored - we do not
want to complicate PID allocations just for the sake of a single old
16-bit interface.]

	Ingo

Re: MAX_PID changes in 2.5.31

[Linus Torvalds]

On Mon, 19 Aug 2002, Ingo Molnar wrote:
> 
> The main problem is that there's the old-style SysV IPC interface that
> uses 16-bit PIDs still. All recent SysV applications (linked against glibc
> 2.2 or newer) use IPC_64, but any application linked against pre-2.2
> glibcs will fail. glibc 2.2 was released 2 years ago, is this enough of a
> timeout to obsolete the non-IPC_64 interfaces?

I actually did the pid changes partly to flush out problems spots, on 
purpose making it 30 bits even though I actually eventually still think 
that we may want to use a few bits for things like node ID numbers etc.

> if that is the case then can i rip all the non-IPC_64 parts out of ipc/*,
> and let non-IPC_64 calls fail? Right now it's silent breakage that
> happens.

Add a warning for now, the same way we did with stat() etc when moving to 
64 bits.

> or, in my threading tree, i introduced a /proc/sys/kernel/pid_max tunable,
> which has the safe conservative value of 32K PIDs, but which can be
> changed by the admin to have higher PIDs.

Fair enough.

		Linus

Re: MAX_PID changes in 2.5.31

[Ingo Molnar]

On Mon, 19 Aug 2002, Linus Torvalds wrote:

> I actually did the pid changes partly to flush out problems spots, on
> purpose making it 30 bits even though I actually eventually still think
> that we may want to use a few bits for things like node ID numbers etc.

lets hope those systems will be 64-bit? But i guess it's too late,
considering the Summit stuff.

> > if that is the case then can i rip all the non-IPC_64 parts out of ipc/*,
> > and let non-IPC_64 calls fail? Right now it's silent breakage that
> > happens.
> 
> Add a warning for now, the same way we did with stat() etc when moving
> to 64 bits.

Ulrich says that glibc 2.2 and upwards already warns about this and is
able to filter all invalid uses of PIDs, so the kernel does not have to
worry about this too much.

> > or, in my threading tree, i introduced a /proc/sys/kernel/pid_max tunable,
> > which has the safe conservative value of 32K PIDs, but which can be
> > changed by the admin to have higher PIDs.
> 
> Fair enough.

(tested) patch attached. It must be used 'with some care' - eg. setting
pid_max to 0 can result in expected behavior, but i dont think we should
overdo this too much.

the patch also increases the pidhash to 8K entries, from 1K entries.

the patch also includes a temporary debugging aid: after starting the
first 300 tasks the kernel jumps PID allocation to max_pid/2 - it does
this once after bootup. This way hopefully we catch more code (and catch
it quicker) than with the previous allocation method.

	Ingo

--- linux/include/linux/sched.h.orig2	Mon Aug 19 22:18:02 2002
+++ linux/include/linux/sched.h	Mon Aug 19 22:18:45 2002
@@ -449,7 +449,7 @@
 extern struct   mm_struct init_mm;
 
 /* PID hashing. (shouldnt this be dynamic?) */
-#define PIDHASH_SZ (4096 >> 2)
+#define PIDHASH_SZ 8192
 extern struct task_struct *pidhash[PIDHASH_SZ];
 
 #define pid_hashfn(x)	((((x) >> 8) ^ (x)) & (PIDHASH_SZ - 1))
--- linux/include/linux/sysctl.h.orig2	Mon Aug 19 22:22:42 2002
+++ linux/include/linux/sysctl.h	Mon Aug 19 22:23:14 2002
@@ -127,6 +127,7 @@
 	KERN_CORE_USES_PID=52,		/* int: use core or core.%pid */
 	KERN_TAINTED=53,	/* int: various kernel tainted flags */
 	KERN_CADPID=54,		/* int: PID of the process to notify on CAD */
+	KERN_PIDMAX=55,		/* int: PID # limit */
 };
 
 
--- linux/kernel/sysctl.c.orig2	Mon Aug 19 22:23:22 2002
+++ linux/kernel/sysctl.c	Mon Aug 19 22:24:08 2002
@@ -51,6 +51,7 @@
 extern int sysrq_enabled;
 extern int core_uses_pid;
 extern int cad_pid;
+extern int pid_max;
 
 /* this is needed for the proc_dointvec_minmax for [fs_]overflow UID and GID */
 static int maxolduid = 65535;
@@ -255,6 +256,8 @@
 	{KERN_S390_USER_DEBUG_LOGGING,"userprocess_debug",
 	 &sysctl_userprocess_debug,sizeof(int),0644,NULL,&proc_dointvec},
 #endif
+	{KERN_PIDMAX, "pid_max", &pid_max, sizeof (int),
+	 0600, NULL, &proc_dointvec},
 	{0}
 };
 
--- linux/kernel/fork.c.orig2	Mon Aug 19 22:13:11 2002
+++ linux/kernel/fork.c	Mon Aug 19 22:30:58 2002
@@ -46,6 +46,14 @@
 
 int max_threads;
 unsigned long total_forks;	/* Handle normal Linux uptimes. */
+
+/*
+ * Protects next_safe, last_pid and pid_max:
+ */
+spinlock_t lastpid_lock = SPIN_LOCK_UNLOCKED;
+
+static int next_safe = DEFAULT_PID_MAX;
+int pid_max = DEFAULT_PID_MAX;
 int last_pid;
 
 struct task_struct *pidhash[PIDHASH_SZ];
@@ -151,46 +159,57 @@
 	return tsk;
 }
 
-spinlock_t lastpid_lock = SPIN_LOCK_UNLOCKED;
-
 static int get_pid(unsigned long flags)
 {
-	static int next_safe = PID_MAX;
 	struct task_struct *p;
+	static int once = 1;
 	int pid;
 
 	if (flags & CLONE_IDLETASK)
 		return 0;
 
 	spin_lock(&lastpid_lock);
-	if((++last_pid) & ~PID_MASK) {
+	if (++last_pid > pid_max) {
 		last_pid = 300;		/* Skip daemons etc. */
 		goto inside;
 	}
-	if(last_pid >= next_safe) {
+	/*
+	 * Temporary debugging code: we increase the PID after
+	 * starting the first 300 tasks, to make sure all code
+	 * that breaks due to larger PIDs does indeed break,
+	 * well before kernel 2.6.
+	 */
+	if (unlikely(once)) {
+		if (last_pid > 300) {
+			once = 0;
+			last_pid = pid_max/2;
+		}
+	}
+
+	if (last_pid >= next_safe) {
 inside:
-		next_safe = PID_MAX;
+		next_safe = pid_max;
 		read_lock(&tasklist_lock);
 	repeat:
 		for_each_task(p) {
-			if(p->pid == last_pid	||
+			if (p->pid == last_pid	||
 			   p->pgrp == last_pid	||
 			   p->tgid == last_pid	||
 			   p->session == last_pid) {
-				if(++last_pid >= next_safe) {
-					if(last_pid & ~PID_MASK)
+				if (++last_pid >= next_safe) {
+					if (last_pid >= pid_max)
 						last_pid = 300;
-					next_safe = PID_MAX;
+					next_safe = pid_max;
 				}
 				goto repeat;
 			}
-			if(p->pid > last_pid && next_safe > p->pid)
+			if (p->pid > last_pid && next_safe > p->pid)
 				next_safe = p->pid;
-			if(p->pgrp > last_pid && next_safe > p->pgrp)
+			if (p->pgrp > last_pid && next_safe > p->pgrp)
 				next_safe = p->pgrp;
-			if(p->tgid > last_pid && next_safe > p->tgid)
+			if (p->tgid > last_pid && next_safe > p->tgid)
 				next_safe = p->tgid;
-			if(p->session > last_pid && next_safe > p->session)
+			if (p->session > last_pid && next_safe > p->session)
 				next_safe = p->session;
 		}
 		read_unlock(&tasklist_lock);
@@ -231,7 +250,7 @@
 		struct file *file;
 
 		retval = -ENOMEM;
-		if(mpnt->vm_flags & VM_DONTCOPY)
+		if (mpnt->vm_flags & VM_DONTCOPY)
 			continue;
 		if (mpnt->vm_flags & VM_ACCOUNT) {
 			unsigned int len = (mpnt->vm_end - mpnt->vm_start) >> PAGE_SHIFT;
@@ -944,12 +963,12 @@
 	vm_area_cachep = kmem_cache_create("vm_area_struct",
 			sizeof(struct vm_area_struct), 0,
 			SLAB_HWCACHE_ALIGN, NULL, NULL);
-	if(!vm_area_cachep)
+	if (!vm_area_cachep)
 		panic("vma_init: Cannot alloc vm_area_struct SLAB cache");
 
 	mm_cachep = kmem_cache_create("mm_struct",
 			sizeof(struct mm_struct), 0,
 			SLAB_HWCACHE_ALIGN, NULL, NULL);
-	if(!mm_cachep)
+	if (!mm_cachep)
 		panic("vma_init: Cannot alloc mm_struct SLAB cache");
 }

Re: MAX_PID changes in 2.5.31

[Richard Gooch]

Ingo Molnar writes:
> 
> Linus,
> 
> afaics, you did the PID_MAX changes in v2.5.31? This is a change i had for
> (surprise) threading purposes already, but done a bit differently.
> 
> The main problem is that there's the old-style SysV IPC interface
> that uses 16-bit PIDs still. All recent SysV applications (linked
> against glibc 2.2 or newer) use IPC_64, but any application linked
> against pre-2.2 glibcs will fail. glibc 2.2 was released 2 years
> ago, is this enough of a timeout to obsolete the non-IPC_64
> interfaces?

Are you saying that people running libc 5 or even glibc 2.1 will
suddenly have their code broken?

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: MAX_PID changes in 2.5.31

[Ingo Molnar]

On Mon, 19 Aug 2002, Richard Gooch wrote:

> Are you saying that people running libc 5 or even glibc 2.1 will
> suddenly have their code broken?

nope. Only if they use the 16-bit PID stuff in SysV IPC semaphores and
message queues.

	Ingo

Re: MAX_PID changes in 2.5.31

[Richard Gooch]

Ingo Molnar writes:
> 
> On Mon, 19 Aug 2002, Richard Gooch wrote:
> 
> > Are you saying that people running libc 5 or even glibc 2.1 will
> > suddenly have their code broken?
> 
> nope. Only if they use the 16-bit PID stuff in SysV IPC semaphores
> and message queues.

In other words, "yes, unless you happen not to use SysV IPC semaphores
or message queues in any of your binaries on your system". So you want
to break binary compatibility. Please don't. I don't want to downgrade
to glibc.

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: MAX_PID changes in 2.5.31

[Robert Love]

On Mon, 2002-08-19 at 18:36, Richard Gooch wrote:

> In other words, "yes, unless you happen not to use SysV IPC semaphores
> or message queues in any of your binaries on your system". So you want
> to break binary compatibility. Please don't. I don't want to downgrade
> to glibc.

So are you saying we can never deprecate interfaces, just so you can
continue using libc5?

Seems saner to keep libc5 in sync with the kernel than vice versa..

	Robert Love

Re: MAX_PID changes in 2.5.31

[Ingo Molnar]

On 19 Aug 2002, Robert Love wrote:

> So are you saying we can never deprecate interfaces, just so you can
> continue using libc5?
> 
> Seems saner to keep libc5 in sync with the kernel than vice versa..

put differently: if you insist on using libc5 (and its tons of security
holes) then you might as well use an older kernel such as 2.4 or 2.6.

	Ingo

Re: MAX_PID changes in 2.5.31

[Richard Gooch]

Robert Love writes:
> On Mon, 2002-08-19 at 18:36, Richard Gooch wrote:
> 
> > In other words, "yes, unless you happen not to use SysV IPC semaphores
> > or message queues in any of your binaries on your system". So you want
> > to break binary compatibility. Please don't. I don't want to downgrade
> > to glibc.
> 
> So are you saying we can never deprecate interfaces, just so you can
> continue using libc5?

Having a stable ABI to user-space has been one of the strong points of
Linux, versus just about everything else out there. And I wouldn't be
using libc 5 if glibc wasn't growing exponentially with time (both in
terms of total bytes and in terms of number of shared libraries I have
to add to my link line).

> Seems saner to keep libc5 in sync with the kernel than vice versa..

It's not just me. There are people using libc 4.

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: MAX_PID changes in 2.5.31

[Ingo Molnar]

On Mon, 19 Aug 2002, Richard Gooch wrote:

> Having a stable ABI to user-space has been one of the strong points of
> Linux, versus just about everything else out there. And I wouldn't be
> using libc 5 if glibc wasn't growing exponentially with time (both in
> terms of total bytes and in terms of number of shared libraries I have
> to add to my link line).

Richard, if you have read the current thread you are replying to, then you
sure must have noticed that i was mailing exactly because i was worried
about binary compatibility with certain old applications, not because i
wanted to break or obsolete them. A patch with a solution was posted.

Since glibc has not linked any application with the old interfaces in the
past 2 years or more, there's no realistic chance to have any 'recent'
applications that will just break. And old stuff has to be careful anyway
=> one more thing to watch out for to get a working system with an old
libc under a new kernel. If you really really need the old app then you
can change pid_max, or you can even run the application under UML running
an older Linux kernel or a new Linux kernel with a more conservative
pid_max value. Plus, as the icing on the cake, new glibc detects old
interface usage safely, so it's not as if things broke silently.

now on to the interface you complain about. It's horribly broken. It
assumes 16-bit PIDs and provides no safeguards against PID overflow. The
new kernel might as well have scrapped the whole 16-bit implementation of
the API and return -ENOSYS on it. Now it provides a way to limit the PID
space to save these applications that use this broken 16-bit API. Good
enough?

	Ingo

Re: MAX_PID changes in 2.5.31

[Thunder from the hill]

Hi,

On Mon, 19 Aug 2002, Richard Gooch wrote:
> It's not just me. There are people using libc 4.

Here!

But I'm not using it for kernels > 2.0. (Well, uClibc and co. do the main 
there.)

			Thunder
-- 
--./../...-/. -.--/---/..-/.-./..././.-../..-. .---/..-/.../- .-
--/../-./..-/-/./--..-- ../.----./.-../.-.. --./../...-/. -.--/---/..-
.- -/---/--/---/.-./.-./---/.--/.-.-.-
--./.-/-.../.-./.././.-../.-.-.-

Re: MAX_PID changes in 2.5.31

[Ingo Molnar]

On Mon, 19 Aug 2002, Richard Gooch wrote:

> > nope. Only if they use the 16-bit PID stuff in SysV IPC semaphores
> > and message queues.
> 
> In other words, "yes, unless you happen not to use SysV IPC semaphores
> or message queues in any of your binaries on your system".

no, in other words: "yes, if you use SysV IPC semaphores/semaphores in any
of your binaries in your system, which binaries were linked against glibc
2.1 or older, and if you have set /proc/sys/kernel/pid_max to a value
higher than 32K."

	Ingo

Re: MAX_PID changes in 2.5.31

[Richard Gooch]

Ingo Molnar writes:
> 
> On Mon, 19 Aug 2002, Richard Gooch wrote:
> 
> > > nope. Only if they use the 16-bit PID stuff in SysV IPC semaphores
> > > and message queues.
> > 
> > In other words, "yes, unless you happen not to use SysV IPC semaphores
> > or message queues in any of your binaries on your system".
> 
> no, in other words: "yes, if you use SysV IPC semaphores/semaphores
> in any of your binaries in your system, which binaries were linked
> against glibc 2.1 or older, and if you have set
> /proc/sys/kernel/pid_max to a value higher than 32K."

Ah, OK. So if I leave /proc/sys/kernel/pid_max alone, nothing will
break. Will the default ever change, or do you plan on leaving it as
is?

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: MAX_PID changes in 2.5.31

[Ingo Molnar]

On Mon, 19 Aug 2002, Richard Gooch wrote:

> > no, in other words: "yes, if you use SysV IPC semaphores/semaphores
> > in any of your binaries in your system, which binaries were linked
> > against glibc 2.1 or older, and if you have set
> > /proc/sys/kernel/pid_max to a value higher than 32K."
> 
> Ah, OK. So if I leave /proc/sys/kernel/pid_max alone, nothing will
> break. Will the default ever change, or do you plan on leaving it as is?

the default will be something sane, ie. probably larger than 32K. There is
safe code in glibc to catch your old applications from doing any
stupidity, then you can change pid_max or just upgrade your system (like
you did with the kernel). It's not a big dependency to require the setting
of pid_max if you want binary compatibility with rare old stuff.

	Ingo

Re: MAX_PID changes in 2.5.31

[H. Peter Anvin]

Followup to:  <200208192242.g7JMgmD29241@vindaloo.ras.ucalgary.ca>
By author:    Richard Gooch <rgooch@ras.ucalgary.ca>
In newsgroup: linux.dev.kernel
> 
> Ah, OK. So if I leave /proc/sys/kernel/pid_max alone, nothing will
> break. Will the default ever change, or do you plan on leaving it as
> is?
> 

It probably should change at some point.  I would favour changing the
default to aggressive in 2.5, to smoke out bugs, and perhaps turn it
back to conservative in 2.6.  In 2.7, we probably should turn on
aggressive for good.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: MAX_PID changes in 2.5.31

[Alan Cox]

On Mon, 2002-08-19 at 23:51, H. Peter Anvin wrote:
> It probably should change at some point.  I would favour changing the
> default to aggressive in 2.5, to smoke out bugs, and perhaps turn it
> back to conservative in 2.6.  In 2.7, we probably should turn on
> aggressive for good.

By that point I'd hope anyone running large workloads is running a 64bit
CPU not x86_32 8) At which point the problem is mostly moot

Re: MAX_PID changes in 2.5.31

[Alan Cox]

On Mon, 2002-08-19 at 23:34, Ingo Molnar wrote:
> 
> On Mon, 19 Aug 2002, Richard Gooch wrote:
> 
> > Are you saying that people running libc 5 or even glibc 2.1 will
> > suddenly have their code broken?
> 
> nope. Only if they use the 16-bit PID stuff in SysV IPC semaphores and
> message queues.

libc5 is very much 16bit pid throughout. It would make sense that our
default (proc settable) pid max is 30000 still so that it only breaks
stuff if you increase it

Re: MAX_PID changes in 2.5.31

[Ingo Molnar]

On 20 Aug 2002, Alan Cox wrote:

> libc5 is very much 16bit pid throughout. It would make sense that our
> default (proc settable) pid max is 30000 still so that it only breaks
> stuff if you increase it

We can have the safe low value in 2.6 i think - it's not that the typical
2.6 kernel is expected to run tens of thousands of tasks.

	Ingo

Re: MAX_PID changes in 2.5.31

[Andries Brouwer]

On Tue, Aug 20, 2002 at 12:29:11AM +0100, Alan Cox wrote:

> libc5 is very much 16bit pid throughout.

Can you clarify?

Andries


#define _IO_pid_t _G_pid_t
typedef int _G_pid_t;
typedef int __pid_t;

Re: MAX_PID changes in 2.5.31

[Alan Cox]

On Tue, 2002-08-20 at 01:33, Andries Brouwer wrote:
> On Tue, Aug 20, 2002 at 12:29:11AM +0100, Alan Cox wrote:
> 
> > libc5 is very much 16bit pid throughout.
> 
> Can you clarify?

It uses the 16bit assuming syscall entry points, and has no knowledge of
the 32bit pid stuff when its needed

Re: MAX_PID changes in 2.5.31

[Andries Brouwer]

> > On Tue, Aug 20, 2002 at 12:29:11AM +0100, Alan Cox wrote:
> > 
> > > libc5 is very much 16bit pid throughout.
> > 
> > Can you clarify?
> 
> It uses the 16bit assuming syscall entry points, and has no knowledge of
> the 32bit pid stuff when its needed

Ha, Alan - I am a bit slow, and you are a bit brief, but let us see.

I interpreted your "throughout" as "also outside IPC", and hence asked
for clarification. For the moment, let me assume that we are just talking
about SYSV IPC. (You were not thinking about uids instead of pids?)

The kernel has these days new structs shmid64_ds, msqid64_ds
and old structs shmid_ds, msqid_ds.
The system calls sys_shmctl(), sys_msgctl() know what struct to use
by checking whether the caller has set the IPC_64 bit.
(No different entry points.) The assignment is done blindly:
	out.shm_cpid    = in->shm_cpid;
hence will truncate pids.

Today glibc has structs shmid_ds and msqid_ds with int ..pid,
and structs old_shmid_ds and old_msqid_ds with
__ipc_pid_t msg_lspid, msg_lrpid, shm_cpid, shm_lpid.
It uses the new structs and sets IPC_64 when that works.
Otherwise it uses old structs, and warns in case something
was truncated.

So, with a modern glibc all is fine.
Here modern includes glibc 2.1.91, but not glibc 2.1.3.

Earlier glibc, and libc5, don't know about IPC_64 and hence use the
old structures.

Remains the question whether that is bad.
With large pid_t, the four variables msg_lspid, msg_lrpid, shm_cpid,
shm_lpid will be truncated.

Who uses these? Nobody, as far as I can see from a recent collection
of RPMs.

So, it would be interesting to see what I overlooked.
Where are the programs that need these variables?

Andries

Re: MAX_PID changes in 2.5.31

[Alan Cox]

On Thu, 2002-08-22 at 23:11, Andries Brouwer wrote:
> Ha, Alan - I am a bit slow, and you are a bit brief, but let us see.
> 
> I interpreted your "throughout" as "also outside IPC", and hence asked
> for clarification. For the moment, let me assume that we are just talking
> about SYSV IPC. (You were not thinking about uids instead of pids?)

I don't think there are any other pid ones. I've no idea what libc4 used
but I don't think I actually care . 

On the kernel side I found a ushort pid in coda, but I can't actually
easily tell if thats a process id or a coda thingy. We have some other
sloppy pid users but they all appear to be int (eg vt_kern.h)


> Remains the question whether that is bad.
> With large pid_t, the four variables msg_lspid, msg_lrpid, shm_cpid,
> shm_lpid will be truncated.
> 
> Who uses these? Nobody, as far as I can see from a recent collection
> of RPMs.

I have to admit I've not looked. I know of one (older AberMUD) which did
it for security tricks but thats hardly relevant to this nor a reason to
worry.

Alan

Re: MAX_PID changes in 2.5.31

[Jan Harkes]

On Thu, Aug 22, 2002 at 11:27:46PM +0100, Alan Cox wrote:
> On the kernel side I found a ushort pid in coda, but I can't actually
> easily tell if thats a process id or a coda thingy. We have some other
> sloppy pid users but they all appear to be int (eg vt_kern.h)

It's both a process id and a Coda thingy :) It should have been pid_t,
but if you change it in the kernel it will break precompiled binaries in
userspace which still assume it is a 16-bit value. It is not used too
often, only when a process is forked off for conflict resolution.

Luckily, I'm getting close to finalizing a new version of Coda's
userspace which has support for realms (or cells, or whatever other
distributed filesystems like to name it) that already has to break
the kernel-venus interface. So I'll add this to my list of incompatible
changes that can go into the patch.

Jan