[Qemu-devel] pthread_create failed: Resource temporarily unavailable

[Qemu-devel] pthread_create failed: Resource temporarily unavailable

[Christoffer Dall]

Hi,

I am seeing some brutal crashes with QEMU running virtio using KVM/ARM.

The culprit seems to be when the process runs out of virtual address
space on 32-bit systems due to some subsystem (virtio?) creating a
large number of pthreads under heavy workloads.

Unfortunately my QEMU expertise is too limited to pin-point the exact
fix, nor do I have resources right now to go into it, but I wanted to
raise this issue and spread general awareness.

Is this a known issue or something that needs to be tracked/documented at least?

Thanks for you input.

-Christoffer

Re: [Qemu-devel] [kvmarm] pthread_create failed: Resource temporarily unavailable

[Adrian Taylor]

[-- Attachment #1: Type: text/plain, Size: 807 bytes --]

Hi Christoffer,

On Wed, Jan 2, 2013 at 7:17 PM, Christoffer Dall <cdall@cs.columbia.edu>wrote:

>
> The culprit seems to be when the process runs out of virtual address
> space on 32-bit systems due to some subsystem (virtio?) creating a
> large number of pthreads under heavy workloads.
>
>
One possibility - qemu uses coroutines for IO. There are various backends
for coroutines, and the qemu configure script will try to choose the best
one. Perhaps your environment is lacking system calls such as makecontext
and the configure script is falling back to a coroutine backend using
pthreads. I've found that this can cause 50000+ threads to be spawned just
during a loadvm operation, for instance.

If this fits the symptoms, try using the --with-coroutine option during
qemu configure.

Regards

Adrian

[-- Attachment #2: Type: text/html, Size: 1288 bytes --]

Re: [Qemu-devel] [kvmarm] pthread_create failed: Resource temporarily unavailable

[Peter Maydell]

On 3 January 2013 00:45, Adrian Taylor <adrian@bromium.com> wrote:
> One possibility - qemu uses coroutines for IO. There are various backends
> for coroutines, and the qemu configure script will try to choose the best
> one. Perhaps your environment is lacking system calls such as makecontext
> and the configure script is falling back to a coroutine backend using
> pthreads.

ARM Linux should support the default 'ucontext' backend but I haven't
checked recently. You can check by looking at the output of configure:
at the end when it prints its findings there is a line
"coroutine backend $whatever". 'ucontext' is good, 'gthread' is bad :-)

-- PMM

Re: [Qemu-devel] pthread_create failed: Resource temporarily unavailable

[Stefan Hajnoczi]

On Wed, Jan 2, 2013 at 8:17 PM, Christoffer Dall <cdall@cs.columbia.edu> wrote:
> I am seeing some brutal crashes with QEMU running virtio using KVM/ARM.
>
> The culprit seems to be when the process runs out of virtual address
> space on 32-bit systems due to some subsystem (virtio?) creating a
> large number of pthreads under heavy workloads.

It would be interesting to see the resource utilization at the point
of failure - virtual memory, number of threads, etc.

> Unfortunately my QEMU expertise is too limited to pin-point the exact
> fix, nor do I have resources right now to go into it, but I wanted to
> raise this issue and spread general awareness.
>
> Is this a known issue or something that needs to be tracked/documented at least?

If you launch QEMU under gdb it will break when qemu_thread_create()
fails to create a new thread.

This may be coming from thread-pool.c:do_spawn_thread().  It shouldn't
be a fatal error since eventually an existing worker thread will
process the queued work.

Please post the backtrace so it's clear where QEMU hits the failure.

Stefan

Re: [Qemu-devel] pthread_create failed: Resource temporarily unavailable

[Andreas Färber]

Hi,

Am 02.01.2013 20:17, schrieb Christoffer Dall:
> I am seeing some brutal crashes with QEMU running virtio using KVM/ARM.
> 
> The culprit seems to be when the process runs out of virtual address
> space on 32-bit systems due to some subsystem (virtio?) creating a
> large number of pthreads under heavy workloads.
> 
> Unfortunately my QEMU expertise is too limited to pin-point the exact
> fix, nor do I have resources right now to go into it, but I wanted to
> raise this issue and spread general awareness.
> 
> Is this a known issue or something that needs to be tracked/documented at least?

It is a known issue that I reported long ago, but there have been higher
priorities. ;)
Note that this failure is not specifically about creating threads but
about thread creation being one of several operations requiring some
free virtual mem, it might as well be a malloc another run. If you trace
memory usage of the process you will likely see dramatic changes over
time... why exactly is still to be investigated.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] pthread_create failed: Resource temporarily unavailable

[Christoffer Dall]

On Thu, Jan 3, 2013 at 6:50 AM, Andreas Färber <afaerber@suse.de> wrote:
>
>> The culprit seems to be when the process runs out of virtual address
>> space on 32-bit systems due to some subsystem (virtio?) creating a
>> large number of pthreads under heavy workloads.
>>
>> Unfortunately my QEMU expertise is too limited to pin-point the exact
>> fix, nor do I have resources right now to go into it, but I wanted to
>> raise this issue and spread general awareness.
>>
>> Is this a known issue or something that needs to be tracked/documented at least?
>
> It is a known issue that I reported long ago, but there have been higher
> priorities. ;)
> Note that this failure is not specifically about creating threads but
> about thread creation being one of severa

hmmm, tried to look at the output of configure, and it does give me this:

coroutine backend ucontext


running qemu-system-arm in GDB outputs this when doing disk IO (like
untar'ing a kernel tree):

[New Thread 0x5045b470 (LWP 6184)]
[New Thread 0x4f148470 (LWP 6185)]
[New Thread 0x4e6ff470 (LWP 6186)]
[New Thread 0x49af5470 (LWP 6187)]
[New Thread 0x492f5470 (LWP 6188)]
[New Thread 0x48af5470 (LWP 6189)]
[New Thread 0x482f5470 (LWP 6190)]
[New Thread 0x47af5470 (LWP 6191)]
[New Thread 0x472f5470 (LWP 6192)]
[New Thread 0x46af5470 (LWP 6193)]
[New Thread 0x462f5470 (LWP 6194)]
[New Thread 0x45af5470 (LWP 6195)]
[New Thread 0x452f5470 (LWP 6196)]
[New Thread 0x44af5470 (LWP 6197)]
[New Thread 0x442f5470 (LWP 6198)]
[New Thread 0x43af5470 (LWP 6199)]
[New Thread 0x432f5470 (LWP 6200)]
[New Thread 0x42af5470 (LWP 6201)]
[New Thread 0x422f5470 (LWP 6202)]
[New Thread 0x41af5470 (LWP 6203)]
[New Thread 0x412f5470 (LWP 6204)]
[New Thread 0x40af5470 (LWP 6205)]
[New Thread 0x402f5470 (LWP 6206)]
[New Thread 0x3faf5470 (LWP 6207)]
[New Thread 0x3f2f5470 (LWP 6208)]
[New Thread 0x3eaf5470 (LWP 6209)]
[New Thread 0x3e2f5470 (LWP 6210)]
[New Thread 0x3daf5470 (LWP 6211)]
[New Thread 0x3d2f5470 (LWP 6212)]
[New Thread 0x3caf5470 (LWP 6213)]
[New Thread 0x3c2f5470 (LWP 6214)]
[New Thread 0x3baf5470 (LWP 6215)]
[New Thread 0x3b2f5470 (LWP 6216)]
[New Thread 0x3aaf5470 (LWP 6217)]
[New Thread 0x3a2f5470 (LWP 6218)]
[New Thread 0x39af5470 (LWP 6219)]
[New Thread 0x392f5470 (LWP 6220)]
[New Thread 0x38af5470 (LWP 6221)]
[New Thread 0x380ff470 (LWP 6222)]
[New Thread 0x378ff470 (LWP 6223)]
[New Thread 0x366f7470 (LWP 6224)]
[New Thread 0x339d2470 (LWP 6225)]
[New Thread 0x331d2470 (LWP 6226)]
[New Thread 0x36eff470 (LWP 6227)]
[New Thread 0x35ef7470 (LWP 6228)]
[New Thread 0x356f7470 (LWP 6229)]
[New Thread 0x329d2470 (LWP 6230)]
[New Thread 0x321d2470 (LWP 6231)]
[New Thread 0x4bff9470 (LWP 6232)]
[New Thread 0x349f2470 (LWP 6234)]
[New Thread 0x305be470 (LWP 6235)]
[New Thread 0x2fdbe470 (LWP 6236)]
[New Thread 0x2f5be470 (LWP 6237)]
[New Thread 0x4afe9470 (LWP 6238)]
[New Thread 0x2edbe470 (LWP 6239)]
[New Thread 0x2e5be470 (LWP 6240)]
[New Thread 0x2ddbe470 (LWP 6241)]
[New Thread 0x2d5be470 (LWP 6243)]
[New Thread 0x2cdbe470 (LWP 6244)]
[Thread 0x442f5470 (LWP 6198) exited]
[Thread 0x4f948470 (LWP 6173) exited]
[Thread 0x3e2f5470 (LWP 6210) exited]
[Thread 0x35ef7470 (LWP 6228) exited]
[Thread 0x452f5470 (LWP 6196) exited]
[Thread 0x51d5c470 (LWP 6171) exited]
[Thread 0x462f5470 (LWP 6194) exited]
[Thread 0x2fdbe470 (LWP 6236) exited]
[Thread 0x2edbe470 (LWP 6239) exited]
[Thread 0x356f7470 (LWP 6229) exited]
[Thread 0x482f5470 (LWP 6190) exited]
[Thread 0x45af5470 (LWP 6195) exited]
[Thread 0x4bff9470 (LWP 6232) exited]
[Thread 0x36eff470 (LWP 6227) exited]
[Thread 0x2ddbe470 (LWP 6241) exited]
[Thread 0x4afe9470 (LWP 6238) exited]
[Thread 0x305be470 (LWP 6235) exited]
[Thread 0x5045b470 (LWP 6184) exited]
[Thread 0x339d2470 (LWP 6225) exited]
[Thread 0x3baf5470 (LWP 6215) exited]
[Thread 0x47af5470 (LWP 6191) exited]
[Thread 0x3faf5470 (LWP 6207) exited]
[Thread 0x3d2f5470 (LWP 6212) exited]
[Thread 0x349f2470 (LWP 6234) exited]
[Thread 0x46af5470 (LWP 6193) exited]
[Thread 0x76c27470 (LWP 6168) exited]
[Thread 0x412f5470 (LWP 6204) exited]
[Thread 0x49af5470 (LWP 6187) exited]
[Thread 0x432f5470 (LWP 6200) exited]
[Thread 0x4f148470 (LWP 6185) exited]
[Thread 0x472f5470 (LWP 6192) exited]
[Thread 0x422f5470 (LWP 6202) exited]
[Thread 0x5145b470 (LWP 6172) exited]
[Thread 0x3b2f5470 (LWP 6216) exited]
[Thread 0x43af5470 (LWP 6199) exited]
[Thread 0x2e5be470 (LWP 6240) exited]
[Thread 0x366f7470 (LWP 6224) exited]
[Thread 0x378ff470 (LWP 6223) exited]
[Thread 0x392f5470 (LWP 6220) exited]
[Thread 0x331d2470 (LWP 6226) exited]
[Thread 0x402f5470 (LWP 6206) exited]
[Thread 0x3f2f5470 (LWP 6208) exited]
[Thread 0x50c5b470 (LWP 6178) exited]
[Thread 0x3caf5470 (LWP 6213) exited]
[Thread 0x2f5be470 (LWP 6237) exited]
[Thread 0x3eaf5470 (LWP 6209) exited]
[Thread 0x3aaf5470 (LWP 6217) exited]
[Thread 0x48af5470 (LWP 6189) exited]
[Thread 0x2cdbe470 (LWP 6244) exited]
[Thread 0x3daf5470 (LWP 6211) exited]
[Thread 0x380ff470 (LWP 6222) exited]
[Thread 0x3c2f5470 (LWP 6214) exited]
[Thread 0x38af5470 (LWP 6221) exited]
[Thread 0x329d2470 (LWP 6230) exited]
[Thread 0x2d5be470 (LWP 6243) exited]
[Thread 0x44af5470 (LWP 6197) exited]
[Thread 0x39af5470 (LWP 6219) exited]
[Thread 0x42af5470 (LWP 6201) exited]
[Thread 0x41af5470 (LWP 6203) exited]
[Thread 0x321d2470 (LWP 6231) exited]
[Thread 0x3a2f5470 (LWP 6218) exited]
[Thread 0x4e6ff470 (LWP 6186) exited]
[Thread 0x40af5470 (LWP 6205) exited]
[Thread 0x492f5470 (LWP 6188) exited]

and then it simply exits, gdb including :(

-Christoffer

Re: [Qemu-devel] pthread_create failed: Resource temporarily unavailable

[Stefan Hajnoczi]

On Thu, Jan 03, 2013 at 01:53:20PM -0500, Christoffer Dall wrote:
> On Thu, Jan 3, 2013 at 6:50 AM, Andreas Färber <afaerber@suse.de> wrote:
> >
> >> The culprit seems to be when the process runs out of virtual address
> >> space on 32-bit systems due to some subsystem (virtio?) creating a
> >> large number of pthreads under heavy workloads.
> >>
> >> Unfortunately my QEMU expertise is too limited to pin-point the exact
> >> fix, nor do I have resources right now to go into it, but I wanted to
> >> raise this issue and spread general awareness.
> >>
> >> Is this a known issue or something that needs to be tracked/documented at least?
> >
> > It is a known issue that I reported long ago, but there have been higher
> > priorities. ;)
> > Note that this failure is not specifically about creating threads but
> > about thread creation being one of severa
> 
> hmmm, tried to look at the output of configure, and it does give me this:
> 
> coroutine backend ucontext
> 
> 
> running qemu-system-arm in GDB outputs this when doing disk IO (like
> untar'ing a kernel tree):
> 
> [New Thread 0x5045b470 (LWP 6184)]
> [New Thread 0x4f148470 (LWP 6185)]
> [New Thread 0x4e6ff470 (LWP 6186)]
> [New Thread 0x49af5470 (LWP 6187)]
> [New Thread 0x492f5470 (LWP 6188)]
> [New Thread 0x48af5470 (LWP 6189)]
> [New Thread 0x482f5470 (LWP 6190)]
> [New Thread 0x47af5470 (LWP 6191)]
> [New Thread 0x472f5470 (LWP 6192)]
> [New Thread 0x46af5470 (LWP 6193)]
> [New Thread 0x462f5470 (LWP 6194)]
> [New Thread 0x45af5470 (LWP 6195)]
> [New Thread 0x452f5470 (LWP 6196)]
> [New Thread 0x44af5470 (LWP 6197)]
> [New Thread 0x442f5470 (LWP 6198)]
> [New Thread 0x43af5470 (LWP 6199)]
> [New Thread 0x432f5470 (LWP 6200)]
> [New Thread 0x42af5470 (LWP 6201)]
> [New Thread 0x422f5470 (LWP 6202)]
> [New Thread 0x41af5470 (LWP 6203)]
> [New Thread 0x412f5470 (LWP 6204)]
> [New Thread 0x40af5470 (LWP 6205)]
> [New Thread 0x402f5470 (LWP 6206)]
> [New Thread 0x3faf5470 (LWP 6207)]
> [New Thread 0x3f2f5470 (LWP 6208)]
> [New Thread 0x3eaf5470 (LWP 6209)]
> [New Thread 0x3e2f5470 (LWP 6210)]
> [New Thread 0x3daf5470 (LWP 6211)]
> [New Thread 0x3d2f5470 (LWP 6212)]
> [New Thread 0x3caf5470 (LWP 6213)]
> [New Thread 0x3c2f5470 (LWP 6214)]
> [New Thread 0x3baf5470 (LWP 6215)]
> [New Thread 0x3b2f5470 (LWP 6216)]
> [New Thread 0x3aaf5470 (LWP 6217)]
> [New Thread 0x3a2f5470 (LWP 6218)]
> [New Thread 0x39af5470 (LWP 6219)]
> [New Thread 0x392f5470 (LWP 6220)]
> [New Thread 0x38af5470 (LWP 6221)]
> [New Thread 0x380ff470 (LWP 6222)]
> [New Thread 0x378ff470 (LWP 6223)]
> [New Thread 0x366f7470 (LWP 6224)]
> [New Thread 0x339d2470 (LWP 6225)]
> [New Thread 0x331d2470 (LWP 6226)]
> [New Thread 0x36eff470 (LWP 6227)]
> [New Thread 0x35ef7470 (LWP 6228)]
> [New Thread 0x356f7470 (LWP 6229)]
> [New Thread 0x329d2470 (LWP 6230)]
> [New Thread 0x321d2470 (LWP 6231)]
> [New Thread 0x4bff9470 (LWP 6232)]
> [New Thread 0x349f2470 (LWP 6234)]
> [New Thread 0x305be470 (LWP 6235)]
> [New Thread 0x2fdbe470 (LWP 6236)]
> [New Thread 0x2f5be470 (LWP 6237)]
> [New Thread 0x4afe9470 (LWP 6238)]
> [New Thread 0x2edbe470 (LWP 6239)]
> [New Thread 0x2e5be470 (LWP 6240)]
> [New Thread 0x2ddbe470 (LWP 6241)]
> [New Thread 0x2d5be470 (LWP 6243)]
> [New Thread 0x2cdbe470 (LWP 6244)]
> [Thread 0x442f5470 (LWP 6198) exited]
> [Thread 0x4f948470 (LWP 6173) exited]
> [Thread 0x3e2f5470 (LWP 6210) exited]
> [Thread 0x35ef7470 (LWP 6228) exited]
> [Thread 0x452f5470 (LWP 6196) exited]
> [Thread 0x51d5c470 (LWP 6171) exited]
> [Thread 0x462f5470 (LWP 6194) exited]
> [Thread 0x2fdbe470 (LWP 6236) exited]
> [Thread 0x2edbe470 (LWP 6239) exited]
> [Thread 0x356f7470 (LWP 6229) exited]
> [Thread 0x482f5470 (LWP 6190) exited]
> [Thread 0x45af5470 (LWP 6195) exited]
> [Thread 0x4bff9470 (LWP 6232) exited]
> [Thread 0x36eff470 (LWP 6227) exited]
> [Thread 0x2ddbe470 (LWP 6241) exited]
> [Thread 0x4afe9470 (LWP 6238) exited]
> [Thread 0x305be470 (LWP 6235) exited]
> [Thread 0x5045b470 (LWP 6184) exited]
> [Thread 0x339d2470 (LWP 6225) exited]
> [Thread 0x3baf5470 (LWP 6215) exited]
> [Thread 0x47af5470 (LWP 6191) exited]
> [Thread 0x3faf5470 (LWP 6207) exited]
> [Thread 0x3d2f5470 (LWP 6212) exited]
> [Thread 0x349f2470 (LWP 6234) exited]
> [Thread 0x46af5470 (LWP 6193) exited]
> [Thread 0x76c27470 (LWP 6168) exited]
> [Thread 0x412f5470 (LWP 6204) exited]
> [Thread 0x49af5470 (LWP 6187) exited]
> [Thread 0x432f5470 (LWP 6200) exited]
> [Thread 0x4f148470 (LWP 6185) exited]
> [Thread 0x472f5470 (LWP 6192) exited]
> [Thread 0x422f5470 (LWP 6202) exited]
> [Thread 0x5145b470 (LWP 6172) exited]
> [Thread 0x3b2f5470 (LWP 6216) exited]
> [Thread 0x43af5470 (LWP 6199) exited]
> [Thread 0x2e5be470 (LWP 6240) exited]
> [Thread 0x366f7470 (LWP 6224) exited]
> [Thread 0x378ff470 (LWP 6223) exited]
> [Thread 0x392f5470 (LWP 6220) exited]
> [Thread 0x331d2470 (LWP 6226) exited]
> [Thread 0x402f5470 (LWP 6206) exited]
> [Thread 0x3f2f5470 (LWP 6208) exited]
> [Thread 0x50c5b470 (LWP 6178) exited]
> [Thread 0x3caf5470 (LWP 6213) exited]
> [Thread 0x2f5be470 (LWP 6237) exited]
> [Thread 0x3eaf5470 (LWP 6209) exited]
> [Thread 0x3aaf5470 (LWP 6217) exited]
> [Thread 0x48af5470 (LWP 6189) exited]
> [Thread 0x2cdbe470 (LWP 6244) exited]
> [Thread 0x3daf5470 (LWP 6211) exited]
> [Thread 0x380ff470 (LWP 6222) exited]
> [Thread 0x3c2f5470 (LWP 6214) exited]
> [Thread 0x38af5470 (LWP 6221) exited]
> [Thread 0x329d2470 (LWP 6230) exited]
> [Thread 0x2d5be470 (LWP 6243) exited]
> [Thread 0x44af5470 (LWP 6197) exited]
> [Thread 0x39af5470 (LWP 6219) exited]
> [Thread 0x42af5470 (LWP 6201) exited]
> [Thread 0x41af5470 (LWP 6203) exited]
> [Thread 0x321d2470 (LWP 6231) exited]
> [Thread 0x3a2f5470 (LWP 6218) exited]
> [Thread 0x4e6ff470 (LWP 6186) exited]
> [Thread 0x40af5470 (LWP 6205) exited]
> [Thread 0x492f5470 (LWP 6188) exited]
> 
> and then it simply exits, gdb including :(

That's weird but it should coredump on SIGABRT if you set ulimit -c
unlimited.  That way you can inspect the coredump with gdb post-mortem.

Stefan