Re: [Bugme-new] [Bug 11381] New: default shmmax

Re: [Bugme-new] [Bug 11381] New: default shmmax

[Andrew Morton]

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Wed, 20 Aug 2008 05:58:57 -0700 (PDT)
bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=11381
> 
>            Summary: default shmmax
>            Product: Other
>            Version: 2.5
>      KernelVersion: 2.6.26.2
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: enhancement
>           Priority: P1
>          Component: Other
>         AssignedTo: other_other@kernel-bugs.osdl.org
>         ReportedBy: peter_e@gmx.net
> 
> 
> I would like to request that the default shmmax setting be increased or the
> downsides of that be documented.  Allow me to explain.
> 
> I am with the PostgreSQL development team.  PostgreSQL is probably one of the
> few users of large amounts of SysV shared memory.  Users would usually want to
> configure anywhere between 10% and 50% of their physical RAM to be used as
> shared memory, which would translate to something on the order of gigabytes
> nowadays.  One of the uniformly annoying things about setting this up is that
> you need to reconfigure the Linux kernel to allow that.  sysctl is nice and
> all, but it still requires users to learn about operating system and kernel
> details, requires root access, and distros don't handle sysctl uniformly
> either.  Maybe there is even a good reason for that, but I couldn't find it,
> and at least I would like to learn it, so that we can pass that information on
> to our users.
> 
> I did some kernel version archeology and found out that up until kernels 2.2
> the shmmax setting appears to have been restricted by CPU-specific constraints,
> as indicated by the default setting being different across CPUs and being
> defined in an asm header.  The default setting on i386 was increased from 16 MB
> to 32 MB somewhere around 1998 in the kernel 2.0 line, and it remains at 32 MB
> in the latest kernel on all architectures.
> 
> Now one question is whether there is a space or time overhead involved with
> setting a high shmmax limit that isn't actually used.  If so, it would be
> interesting to know what that overhead is.  The feeling I get from browsing the
> kernel source code over time is that there was some management overhead and/or
> some restrictions about this in old kernels, but that nowadays it doesn't
> really seem to matter much anymore.  I suspect instead that this whole thing
> was just forgotten, because few applications use large amounts of shared
> memory.
> 
> So, if you want to do us a favor, could you please see about increasing the
> default shmmax setting to whatever the theoretical maximum is?
> 

I don't think anybody has even thought about the shmmax default in
years.  Sure, it might be time to reexamine that.

It would be useful to get distro input on this.  Do they override the
kernel default at boot time?  If so, what do they do?


Also, from a quick read it looks to me that shmmax is busted in the
non-init namespace.

clone_ipc_ns() calls shm_init_ns() which does

	ns->shm_ctlmax = SHMMAX;

which a) fails to inherit the parent's setting and b) cannot be altered
from SHMMAX via the sysctl?

Re: [Bugme-new] [Bug 11381] New: default shmmax

[Alan Cox]

> It would be useful to get distro input on this.  Do they override the
> kernel default at boot time?  If so, what do they do?

Red Hat provide a sysctl tuning config file and I believe things like the
Oracle install docs cover this.

There is btw no earthly reason why a postgres package can't include a
tool to do this or postgres can't check and update it as part of its
own set up and config file options


Alan

Re: [Bugme-new] [Bug 11381] New: default shmmax

[Peter Eisentraut]

Alan Cox wrote:
> > It would be useful to get distro input on this.  Do they override the
> > kernel default at boot time?  If so, what do they do?
>
> Red Hat provide a sysctl tuning config file and I believe things like the
> Oracle install docs cover this.
>
> There is btw no earthly reason why a postgres package can't include a
> tool to do this or postgres can't check and update it as part of its
> own set up and config file options

I have explored this, it would be possible, but it would be a pain in the 
neck.  You'd need to add an option to postgres to print out how much shared 
memory you would need for a given configuration (because there are multiple 
factors in this), run this across all postgres instances on the machine, sum 
it up, then arrange for this to be fed back to sysctl or pasted back into 
sysctl.conf, being careful not to override or lower an existing setting that 
might have wanted to reserve space for other things.  Note that to make this 
robust you would have to rerun this after every configuration change + 
restart cycle across all postgres instances on the machine, race conditions 
included.  And you'd probably violate a few packaging guidelines on the way.  
Plus, this would have to be distro-specific (not Linux-specific, but 
distro-specific), doesn't help managed hosting with no root access, and 
doesn't help people building from source.  So altogether it is very 
complicated.

In the meantime I am trying to explore why the shmmax default is what it is.  
Or why it has to exist at all.  If we can figure that out, maybe we can solve 
a lot of users a lot of trouble with much less effort (including Oracle 
users, why not).

Re: [Bugme-new] [Bug 11381] New: default shmmax

[adobriyan]

On Wed, Aug 20, 2008 at 12:00:43PM -0700, Andrew Morton wrote:
> 
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Wed, 20 Aug 2008 05:58:57 -0700 (PDT)
> bugme-daemon@bugzilla.kernel.org wrote:
> 
> > http://bugzilla.kernel.org/show_bug.cgi?id=11381
> > 
> >            Summary: default shmmax
> >            Product: Other
> >            Version: 2.5
> >      KernelVersion: 2.6.26.2
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: enhancement
> >           Priority: P1
> >          Component: Other
> >         AssignedTo: other_other@kernel-bugs.osdl.org
> >         ReportedBy: peter_e@gmx.net
> > 
> > 
> > I would like to request that the default shmmax setting be increased or the
> > downsides of that be documented.  Allow me to explain.
> > 
> > I am with the PostgreSQL development team.  PostgreSQL is probably one of the
> > few users of large amounts of SysV shared memory.  Users would usually want to
> > configure anywhere between 10% and 50% of their physical RAM to be used as
> > shared memory, which would translate to something on the order of gigabytes
> > nowadays.  One of the uniformly annoying things about setting this up is that
> > you need to reconfigure the Linux kernel to allow that.  sysctl is nice and
> > all, but it still requires users to learn about operating system and kernel
> > details, requires root access, and distros don't handle sysctl uniformly
> > either.  Maybe there is even a good reason for that, but I couldn't find it,
> > and at least I would like to learn it, so that we can pass that information on
> > to our users.
> > 
> > I did some kernel version archeology and found out that up until kernels 2.2
> > the shmmax setting appears to have been restricted by CPU-specific constraints,
> > as indicated by the default setting being different across CPUs and being
> > defined in an asm header.  The default setting on i386 was increased from 16 MB
> > to 32 MB somewhere around 1998 in the kernel 2.0 line, and it remains at 32 MB
> > in the latest kernel on all architectures.
> > 
> > Now one question is whether there is a space or time overhead involved with
> > setting a high shmmax limit that isn't actually used.  If so, it would be
> > interesting to know what that overhead is.  The feeling I get from browsing the
> > kernel source code over time is that there was some management overhead and/or
> > some restrictions about this in old kernels, but that nowadays it doesn't
> > really seem to matter much anymore.  I suspect instead that this whole thing
> > was just forgotten, because few applications use large amounts of shared
> > memory.
> > 
> > So, if you want to do us a favor, could you please see about increasing the
> > default shmmax setting to whatever the theoretical maximum is?
> > 
> 
> I don't think anybody has even thought about the shmmax default in
> years.  Sure, it might be time to reexamine that.
> 
> It would be useful to get distro input on this.  Do they override the
> kernel default at boot time?  If so, what do they do?
> 
> 
> Also, from a quick read it looks to me that shmmax is busted in the
> non-init namespace.
> 
> clone_ipc_ns() calls shm_init_ns() which does
> 
> 	ns->shm_ctlmax = SHMMAX;
> 
> which a) fails to inherit the parent's setting and

This is debatable if such behaviour should be default, this makes one ipc_ns
more ipc_ns than others.

> b) cannot be altered from SHMMAX via the sysctl?

It can be altered. See kludge called get_ipc().

(well, I haven't checked personally, but it's a bug if IPC sysctls
aren't independently controllable after CLONE_NEWIPC, complain loudly).

Re: [Bugme-new] [Bug 11381] New: default shmmax

[adobriyan]

On Wed, Aug 20, 2008 at 11:12:57PM +0400,  wrote:
> On Wed, Aug 20, 2008 at 12:00:43PM -0700, Andrew Morton wrote:
> > 
> > (switched to email.  Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
> > 
> > On Wed, 20 Aug 2008 05:58:57 -0700 (PDT)
> > bugme-daemon@bugzilla.kernel.org wrote:
> > 
> > > http://bugzilla.kernel.org/show_bug.cgi?id=11381
> > > 
> > >            Summary: default shmmax
> > >            Product: Other
> > >            Version: 2.5
> > >      KernelVersion: 2.6.26.2
> > >           Platform: All
> > >         OS/Version: Linux
> > >               Tree: Mainline
> > >             Status: NEW
> > >           Severity: enhancement
> > >           Priority: P1
> > >          Component: Other
> > >         AssignedTo: other_other@kernel-bugs.osdl.org
> > >         ReportedBy: peter_e@gmx.net
> > > 
> > > 
> > > I would like to request that the default shmmax setting be increased or the
> > > downsides of that be documented.  Allow me to explain.
> > > 
> > > I am with the PostgreSQL development team.  PostgreSQL is probably one of the
> > > few users of large amounts of SysV shared memory.  Users would usually want to
> > > configure anywhere between 10% and 50% of their physical RAM to be used as
> > > shared memory, which would translate to something on the order of gigabytes
> > > nowadays.  One of the uniformly annoying things about setting this up is that
> > > you need to reconfigure the Linux kernel to allow that.  sysctl is nice and
> > > all, but it still requires users to learn about operating system and kernel
> > > details, requires root access, and distros don't handle sysctl uniformly
> > > either.  Maybe there is even a good reason for that, but I couldn't find it,
> > > and at least I would like to learn it, so that we can pass that information on
> > > to our users.
> > > 
> > > I did some kernel version archeology and found out that up until kernels 2.2
> > > the shmmax setting appears to have been restricted by CPU-specific constraints,
> > > as indicated by the default setting being different across CPUs and being
> > > defined in an asm header.  The default setting on i386 was increased from 16 MB
> > > to 32 MB somewhere around 1998 in the kernel 2.0 line, and it remains at 32 MB
> > > in the latest kernel on all architectures.
> > > 
> > > Now one question is whether there is a space or time overhead involved with
> > > setting a high shmmax limit that isn't actually used.  If so, it would be
> > > interesting to know what that overhead is.  The feeling I get from browsing the
> > > kernel source code over time is that there was some management overhead and/or
> > > some restrictions about this in old kernels, but that nowadays it doesn't
> > > really seem to matter much anymore.  I suspect instead that this whole thing
> > > was just forgotten, because few applications use large amounts of shared
> > > memory.
> > > 
> > > So, if you want to do us a favor, could you please see about increasing the
> > > default shmmax setting to whatever the theoretical maximum is?
> > > 
> > 
> > I don't think anybody has even thought about the shmmax default in
> > years.  Sure, it might be time to reexamine that.
> > 
> > It would be useful to get distro input on this.  Do they override the
> > kernel default at boot time?  If so, what do they do?
> > 
> > 
> > Also, from a quick read it looks to me that shmmax is busted in the
> > non-init namespace.
> > 
> > clone_ipc_ns() calls shm_init_ns() which does
> > 
> > 	ns->shm_ctlmax = SHMMAX;
> > 
> > which a) fails to inherit the parent's setting and
> 
> This is debatable if such behaviour should be default, this makes one ipc_ns
> more ipc_ns than others.

Oh, I forgot for a moment, that mainline has hierarchical ipc_ns.

Re: [Bugme-new] [Bug 11381] New: default shmmax

[Solofo.Ramangalahy-6ktuUTfB/bM]

>>>>> On Wed, 20 Aug 2008 12:00:43 -0700, Andrew Morton <akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org> said:
    Andrew>I don't think anybody has even thought about the shmmax default in
    Andrew> years.

IIRC, Matt (explicit CC: added) did a review of IPC tunables and how
they are changed in the distributions.

    Andrew> It would be useful to get distro input on this.  Do they
    Andrew> override the kernel default at boot time?  If so, what do
    Andrew> they do?

OpenSuse (11.0 at least) has this:
-#define SHMMAX 0x2000000                /* max shared seg size (bytes) */
+#define SHMMAX ULONG_MAX                /* max shared seg size (bytes) */

Re: [Bugme-new] [Bug 11381] New: default shmmax

[Peter Eisentraut]

Solofo.Ramangalahy@bull.net wrote:
> OpenSuse (11.0 at least) has this:
> -#define SHMMAX 0x2000000                /* max shared seg size (bytes) */
> +#define SHMMAX ULONG_MAX                /* max shared seg size (bytes) */

Interesting, I don't see that in SLES 10, so it must be quite new.  If no 
problems are found with this, this would be great for the upstream kernel.

Re: [Bugme-new] [Bug 11381] New: default shmmax

[Peter Eisentraut]

Andrew Morton wrote:
> It would be useful to get distro input on this.  Do they override the
> kernel default at boot time?  If so, what do they do?

Debian and Ubuntu have a sysctl.conf that is processed at boot time, but it 
does not contain anything related to shm* by default.

Suse has a sysctl.conf, but it is not processed at boot by default.  You have 
to activate a special boot service in the runlevel system to make it actually 
run at boot.  (Try explaining that to a PostgreSQL user.)  There is nothing 
related to shm* in the default configuration.