[PATCH v5] tools/xenconsoled: Increase file descriptor limit

[PATCH v5] tools/xenconsoled: Increase file descriptor limit

[Andrew Cooper]

XenServer's VM density testing uncovered a regression when moving from
sysvinit to systemd where the file descriptor limit dropped from 4096 to
1024. (XenServer had previously inserted a ulimit statement into its
initscripts.)

One solution is to use LimitNOFILE=4096 in xenconsoled.service to match the
lost ulimit, but that is only a stopgap solution.

As Xenconsoled genuinely needs a large number of file descriptors if a large
number of domains are running, attempt to increase the limit.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>

---
v5:
 * Drop system maximum checking
 * Unify set paths
v4:
 * Calculate fd limit based on domid ABI - result is 132008 fds
 * Warn if sufficient fds are not available.
v3:
 * Hide Linux specific bits in #ifdef __linux__
v2:
 * Always increase soft limit to hard limit
 * Correct commment regarding number of file descriptors
 * long -> unsigned long as that appears to be the underlying type of an rlim_t
---
 tools/console/daemon/main.c |   36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/tools/console/daemon/main.c b/tools/console/daemon/main.c
index 92d2fc4..6e84f5a 100644
--- a/tools/console/daemon/main.c
+++ b/tools/console/daemon/main.c
@@ -26,6 +26,7 @@
 #include <string.h>
 #include <signal.h>
 #include <sys/types.h>
+#include <sys/resource.h>
 
 #include "xenctrl.h"
 
@@ -55,6 +56,39 @@ static void version(char *name)
 	printf("Xen Console Daemon 3.0\n");
 }
 
+static void increase_fd_limit(void)
+{
+	/*
+	 * We require many file descriptors:
+	 * - per domain: pty master, pty slave, logfile and evtchn
+	 * - misc extra: hypervisor log, privcmd, gntdev, std...
+	 *
+	 * Allow a generous 1000 for misc, and calculate the maximum possible
+	 * number of fds which could be used.
+	 */
+	unsigned min_fds = (DOMID_FIRST_RESERVED * 4) + 1000;
+	struct rlimit lim, new = { min_fds, min_fds };
+
+	if (getrlimit(RLIMIT_NOFILE, &lim) < 0) {
+		fprintf(stderr, "Failed to obtain fd limit: %s\n",
+			strerror(errno));
+		exit(1);
+	}
+
+	/* Do we already have sufficient? Great! */
+	if (lim.rlim_cur >= min_fds)
+		return;
+
+	/* Try to increase our limit. */
+	if (setrlimit(RLIMIT_NOFILE, &new) < 0)
+		syslog(LOG_WARNING,
+		       "Unable to increase fd limit from {%lu, %lu} to "
+		       "{%lu, %lu}: (%s) - May run out with lots of domains",
+		       lim.rlim_cur, lim.rlim_max,
+		       new.rlim_cur, new.rlim_max,
+		       strerror(errno));
+}
+
 int main(int argc, char **argv)
 {
 	const char *sopts = "hVvit:o:";
@@ -154,6 +188,8 @@ int main(int argc, char **argv)
 	openlog("xenconsoled", syslog_option, LOG_DAEMON);
 	setlogmask(syslog_mask);
 
+	increase_fd_limit();
+
 	if (!is_interactive) {
 		daemonize(pidfile ? pidfile : "/var/run/xenconsoled.pid");
 	}
-- 
1.7.10.4

Re: [PATCH v5] tools/xenconsoled: Increase file descriptor limit

[Ian Jackson]

Andrew Cooper writes ("[PATCH v5] tools/xenconsoled: Increase file descriptor limit"):
> XenServer's VM density testing uncovered a regression when moving from
> sysvinit to systemd where the file descriptor limit dropped from 4096 to
> 1024. (XenServer had previously inserted a ulimit statement into its
> initscripts.)
...

Thanks, and sorry to be pernickety.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH v5] tools/xenconsoled: Increase file descriptor limit

[Ian Campbell]

On Fri, 2015-02-27 at 17:53 +0000, Andrew Cooper wrote:
> XenServer's VM density testing uncovered a regression when moving from
> sysvinit to systemd where the file descriptor limit dropped from 4096 to
> 1024. (XenServer had previously inserted a ulimit statement into its
> initscripts.)
> 
> One solution is to use LimitNOFILE=4096 in xenconsoled.service to match the
> lost ulimit, but that is only a stopgap solution.
> 
> As Xenconsoled genuinely needs a large number of file descriptors if a large
> number of domains are running, attempt to increase the limit.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> CC: Ian Campbell <Ian.Campbell@citrix.com>
> CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
> CC: Wei Liu <wei.liu2@citrix.com>

I tried to apply but I'm afraid that for 32-bit userspace this gives me:

daemon/main.c: In function 'increase_fd_limit':
daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 3 has type 'rlim_t' [-Werror=format]
daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'rlim_t' [-Werror=format]
daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 5 has type 'rlim_t' [-Werror=format]
daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 6 has type 'rlim_t' [-Werror=format]

I've no idea how one is formally supposed to print and rlim_r.

Ian.

Re: [PATCH v5] tools/xenconsoled: Increase file descriptor limit

[Ian Campbell]

On Mon, 2015-03-02 at 14:22 +0000, Ian Campbell wrote:
> On Fri, 2015-02-27 at 17:53 +0000, Andrew Cooper wrote:
> > XenServer's VM density testing uncovered a regression when moving from
> > sysvinit to systemd where the file descriptor limit dropped from 4096 to
> > 1024. (XenServer had previously inserted a ulimit statement into its
> > initscripts.)
> > 
> > One solution is to use LimitNOFILE=4096 in xenconsoled.service to match the
> > lost ulimit, but that is only a stopgap solution.
> > 
> > As Xenconsoled genuinely needs a large number of file descriptors if a large
> > number of domains are running, attempt to increase the limit.
> > 
> > Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> > CC: Ian Campbell <Ian.Campbell@citrix.com>
> > CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
> > CC: Wei Liu <wei.liu2@citrix.com>
> 
> I tried to apply but I'm afraid that for 32-bit userspace this gives me:
> 
> daemon/main.c: In function 'increase_fd_limit':
> daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 3 has type 'rlim_t' [-Werror=format]
> daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'rlim_t' [-Werror=format]
> daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 5 has type 'rlim_t' [-Werror=format]
> daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 6 has type 'rlim_t' [-Werror=format]
> 
> I've no idea how one is formally supposed to print and rlim_r.

The Internet(tm) seems to think "by casting to long long" is the
answer :-/

Ian.

Re: [PATCH v5] tools/xenconsoled: Increase file descriptor limit

[Andrew Cooper]

On 02/03/15 14:22, Ian Campbell wrote:
> On Fri, 2015-02-27 at 17:53 +0000, Andrew Cooper wrote:
>> XenServer's VM density testing uncovered a regression when moving from
>> sysvinit to systemd where the file descriptor limit dropped from 4096 to
>> 1024. (XenServer had previously inserted a ulimit statement into its
>> initscripts.)
>>
>> One solution is to use LimitNOFILE=4096 in xenconsoled.service to match the
>> lost ulimit, but that is only a stopgap solution.
>>
>> As Xenconsoled genuinely needs a large number of file descriptors if a large
>> number of domains are running, attempt to increase the limit.
>>
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
>> CC: Ian Campbell <Ian.Campbell@citrix.com>
>> CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
>> CC: Wei Liu <wei.liu2@citrix.com>
> I tried to apply but I'm afraid that for 32-bit userspace this gives me:
>
> daemon/main.c: In function 'increase_fd_limit':
> daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 3 has type 'rlim_t' [-Werror=format]
> daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'rlim_t' [-Werror=format]
> daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 5 has type 'rlim_t' [-Werror=format]
> daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 6 has type 'rlim_t' [-Werror=format]
>
> I've no idea how one is formally supposed to print and rlim_r.
>
> Ian.
>

Urgh - that would be why all the examples I found had an explicit
(unsigned long long) cast for the values for printf().

I can respin if you wish.

~Andrew

Re: [PATCH v5] tools/xenconsoled: Increase file descriptor limit

[Ian Campbell]

On Mon, 2015-03-02 at 14:24 +0000, Andrew Cooper wrote:
> On 02/03/15 14:22, Ian Campbell wrote:
> > On Fri, 2015-02-27 at 17:53 +0000, Andrew Cooper wrote:
> >> XenServer's VM density testing uncovered a regression when moving from
> >> sysvinit to systemd where the file descriptor limit dropped from 4096 to
> >> 1024. (XenServer had previously inserted a ulimit statement into its
> >> initscripts.)
> >>
> >> One solution is to use LimitNOFILE=4096 in xenconsoled.service to match the
> >> lost ulimit, but that is only a stopgap solution.
> >>
> >> As Xenconsoled genuinely needs a large number of file descriptors if a large
> >> number of domains are running, attempt to increase the limit.
> >>
> >> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> >> CC: Ian Campbell <Ian.Campbell@citrix.com>
> >> CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
> >> CC: Wei Liu <wei.liu2@citrix.com>
> > I tried to apply but I'm afraid that for 32-bit userspace this gives me:
> >
> > daemon/main.c: In function 'increase_fd_limit':
> > daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 3 has type 'rlim_t' [-Werror=format]
> > daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'rlim_t' [-Werror=format]
> > daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 5 has type 'rlim_t' [-Werror=format]
> > daemon/main.c:89:10: error: format '%lu' expects argument of type 'long unsigned int', but argument 6 has type 'rlim_t' [-Werror=format]
> >
> > I've no idea how one is formally supposed to print and rlim_r.
> >
> > Ian.
> >
> 
> Urgh - that would be why all the examples I found had an explicit
> (unsigned long long) cast for the values for printf().
> 
> I can respin if you wish.

yes, please.

[PATCH v6] tools/xenconsoled: Increase file descriptor limit

[Andrew Cooper]

XenServer's VM density testing uncovered a regression when moving from
sysvinit to systemd where the file descriptor limit dropped from 4096 to
1024. (XenServer had previously inserted a ulimit statement into its
initscripts.)

One solution is to use LimitNOFILE=4096 in xenconsoled.service to match the
lost ulimit, but that is only a stopgap solution.

As Xenconsoled genuinely needs a large number of file descriptors if a large
number of domains are running, attempt to increase the limit.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

---
v6:
 * Fix 32bit build.  (rlimit_t is an arch-specific width)
v5:
 * Drop system maximum checking
 * Unify set paths
v4:
 * Calculate fd limit based on domid ABI - result is 132008 fds
 * Warn if sufficient fds are not available.
v3:
 * Hide Linux specific bits in #ifdef __linux__
v2:
 * Always increase soft limit to hard limit
 * Correct commment regarding number of file descriptors
 * long -> unsigned long as that appears to be the underlying type of an rlim_t
---
 tools/console/daemon/main.c |   38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/tools/console/daemon/main.c b/tools/console/daemon/main.c
index 92d2fc4..11de5c9 100644
--- a/tools/console/daemon/main.c
+++ b/tools/console/daemon/main.c
@@ -26,6 +26,7 @@
 #include <string.h>
 #include <signal.h>
 #include <sys/types.h>
+#include <sys/resource.h>
 
 #include "xenctrl.h"
 
@@ -55,6 +56,41 @@ static void version(char *name)
 	printf("Xen Console Daemon 3.0\n");
 }
 
+static void increase_fd_limit(void)
+{
+	/*
+	 * We require many file descriptors:
+	 * - per domain: pty master, pty slave, logfile and evtchn
+	 * - misc extra: hypervisor log, privcmd, gntdev, std...
+	 *
+	 * Allow a generous 1000 for misc, and calculate the maximum possible
+	 * number of fds which could be used.
+	 */
+	unsigned min_fds = (DOMID_FIRST_RESERVED * 4) + 1000;
+	struct rlimit lim, new = { min_fds, min_fds };
+
+	if (getrlimit(RLIMIT_NOFILE, &lim) < 0) {
+		fprintf(stderr, "Failed to obtain fd limit: %s\n",
+			strerror(errno));
+		exit(1);
+	}
+
+	/* Do we already have sufficient? Great! */
+	if (lim.rlim_cur >= min_fds)
+		return;
+
+	/* Try to increase our limit. */
+	if (setrlimit(RLIMIT_NOFILE, &new) < 0)
+		syslog(LOG_WARNING,
+		       "Unable to increase fd limit from {%llu, %llu} to "
+		       "{%llu, %llu}: (%s) - May run out with lots of domains",
+		       (unsigned long long)lim.rlim_cur,
+		       (unsigned long long)lim.rlim_max,
+		       (unsigned long long)new.rlim_cur,
+		       (unsigned long long)new.rlim_max,
+		       strerror(errno));
+}
+
 int main(int argc, char **argv)
 {
 	const char *sopts = "hVvit:o:";
@@ -154,6 +190,8 @@ int main(int argc, char **argv)
 	openlog("xenconsoled", syslog_option, LOG_DAEMON);
 	setlogmask(syslog_mask);
 
+	increase_fd_limit();
+
 	if (!is_interactive) {
 		daemonize(pidfile ? pidfile : "/var/run/xenconsoled.pid");
 	}
-- 
1.7.10.4

Re: [PATCH v6] tools/xenconsoled: Increase file descriptor limit

[Ian Campbell]

On Mon, 2015-03-02 at 15:04 +0000, Andrew Cooper wrote:
> XenServer's VM density testing uncovered a regression when moving from
> sysvinit to systemd where the file descriptor limit dropped from 4096 to
> 1024. (XenServer had previously inserted a ulimit statement into its
> initscripts.)
> 
> One solution is to use LimitNOFILE=4096 in xenconsoled.service to match the
> lost ulimit, but that is only a stopgap solution.
> 
> As Xenconsoled genuinely needs a large number of file descriptors if a large
> number of domains are running, attempt to increase the limit.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> 
> ---
> v6:
>  * Fix 32bit build.  (rlimit_t is an arch-specific width)

Successfully applied & pushed this time, thanks.

Re: [PATCH v6] tools/xenconsoled: Increase file descriptor limit

[Frediano Ziglio]

2015-03-02 15:04 GMT+00:00 Andrew Cooper <andrew.cooper3@citrix.com>:
> XenServer's VM density testing uncovered a regression when moving from
> sysvinit to systemd where the file descriptor limit dropped from 4096 to
> 1024. (XenServer had previously inserted a ulimit statement into its
> initscripts.)
>
> One solution is to use LimitNOFILE=4096 in xenconsoled.service to match the
> lost ulimit, but that is only a stopgap solution.
>
> As Xenconsoled genuinely needs a large number of file descriptors if a large
> number of domains are running, attempt to increase the limit.
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
>

Nothing against the patch but I think you should then check maximum
number of pty on the system (ie
http://www.cyberciti.biz/tips/howto-linux-increase-pty-session.html).
Usually is high (like 4096) but still less than theory Xen domain
limit (about 32k). This (in theory) would lead to domain0 with a not
working interactive ssh.
I would also be curious why the server always keeps slave descriptor open.

Frediano

> ---
> v6:
>  * Fix 32bit build.  (rlimit_t is an arch-specific width)
> v5:
>  * Drop system maximum checking
>  * Unify set paths
> v4:
>  * Calculate fd limit based on domid ABI - result is 132008 fds
>  * Warn if sufficient fds are not available.
> v3:
>  * Hide Linux specific bits in #ifdef __linux__
> v2:
>  * Always increase soft limit to hard limit
>  * Correct commment regarding number of file descriptors
>  * long -> unsigned long as that appears to be the underlying type of an rlim_t
> ---
>  tools/console/daemon/main.c |   38 ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 38 insertions(+)
>
> diff --git a/tools/console/daemon/main.c b/tools/console/daemon/main.c
> index 92d2fc4..11de5c9 100644
> --- a/tools/console/daemon/main.c
> +++ b/tools/console/daemon/main.c
> @@ -26,6 +26,7 @@
>  #include <string.h>
>  #include <signal.h>
>  #include <sys/types.h>
> +#include <sys/resource.h>
>
>  #include "xenctrl.h"
>
> @@ -55,6 +56,41 @@ static void version(char *name)
>         printf("Xen Console Daemon 3.0\n");
>  }
>
> +static void increase_fd_limit(void)
> +{
> +       /*
> +        * We require many file descriptors:
> +        * - per domain: pty master, pty slave, logfile and evtchn
> +        * - misc extra: hypervisor log, privcmd, gntdev, std...
> +        *
> +        * Allow a generous 1000 for misc, and calculate the maximum possible
> +        * number of fds which could be used.
> +        */
> +       unsigned min_fds = (DOMID_FIRST_RESERVED * 4) + 1000;
> +       struct rlimit lim, new = { min_fds, min_fds };
> +
> +       if (getrlimit(RLIMIT_NOFILE, &lim) < 0) {
> +               fprintf(stderr, "Failed to obtain fd limit: %s\n",
> +                       strerror(errno));
> +               exit(1);
> +       }
> +
> +       /* Do we already have sufficient? Great! */
> +       if (lim.rlim_cur >= min_fds)
> +               return;
> +
> +       /* Try to increase our limit. */
> +       if (setrlimit(RLIMIT_NOFILE, &new) < 0)
> +               syslog(LOG_WARNING,
> +                      "Unable to increase fd limit from {%llu, %llu} to "
> +                      "{%llu, %llu}: (%s) - May run out with lots of domains",
> +                      (unsigned long long)lim.rlim_cur,
> +                      (unsigned long long)lim.rlim_max,
> +                      (unsigned long long)new.rlim_cur,
> +                      (unsigned long long)new.rlim_max,
> +                      strerror(errno));
> +}
> +
>  int main(int argc, char **argv)
>  {
>         const char *sopts = "hVvit:o:";
> @@ -154,6 +190,8 @@ int main(int argc, char **argv)
>         openlog("xenconsoled", syslog_option, LOG_DAEMON);
>         setlogmask(syslog_mask);
>
> +       increase_fd_limit();
> +
>         if (!is_interactive) {
>                 daemonize(pidfile ? pidfile : "/var/run/xenconsoled.pid");
>         }
> --
> 1.7.10.4
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel