* stack size limit issues with xen + qemu + rbd
@ 2016-09-16 20:55 ` Chris Patterson
0 siblings, 0 replies; 5+ messages in thread
From: Chris Patterson @ 2016-09-16 20:55 UTC (permalink / raw)
To: xen-devel, qemu-devel, ceph-devel
I have spent some time investigating a case where qemu is failing to
register xenstore watches for a PV guest once I enable vfb (and
thereby triggering the creation of a qemu instance).
The qemu logs show something along the lines of:
xen be core: xen be core: xen be: watching backend path
(backend/console/3) failed
xen be: watching backend path (backend/console/3) failed
xen be core: xen be core: xen be: watching backend path (backend/vkbd/3) failed
xen be: watching backend path (backend/vkbd/3) failed
xen be core: xen be core: xen be: watching backend path (backend/qdisk/3) failed
xen be: watching backend path (backend/qdisk/3) failed
xen be core: xen be core: xen be: watching backend path (backend/qusb/3) failed
xen be: watching backend path (backend/qusb/3) failed
xen be core: xen be core: xen be: watching backend path (backend/vfb/3) failed
xen be: watching backend path (backend/vfb/3) failed
xen be core: xen be core: xen be: watching backend path (backend/qnic/3) failed
xen be: watching backend path (backend/qnic/3) failed
I have tested qemu master, qemu-xen in the master xen tree, as well as
a few tags all with the same issue.
I came across a similar issue reported by Juergen Gross:
https://lists.nongnu.org/archive/html/qemu-devel/2016-07/msg03341.html
Sure enough, the thread stack size was the culprit. I had been
testing with qemu with the associated fix "vnc-tight: fix regression
with libxenstore" as it is in master, so that wasn't it...
I did some basic analysis of the qemu binary and the libraries it is pulling in:
for lib in $(ldd /usr/local/bin/qemu-system-i386 | grep -o '/.* '); do
echo "lib=$lib"; readelf -S "$lib" | grep -e tbss -e tdata -A1 ; done
The largest consumers were:
lib=/usr/lib/x86_64-linux-gnu/librbd.so.1
[17] .tbss NOBITS 000000000088fed0 0068fed0
0000000000001820 0000000000000000 WAT 0 0 8
lib=/usr/lib/x86_64-linux-gnu/librados.so.2
[17] .tbss NOBITS 0000000000718600 00518600
0000000000000aa0 0000000000000000 WAT 0 0 8
IIUC, librbd & librados are using nearly 9k of the 16k alone (I am
assuming this thread-local storage must be consumed as part of the
thread's stack)?
I narrowed that down to Ceph's usage of __thread in stringify() in
src/include/stringify.h.
To make things functional, the options were either to:
(a) disable rbd at configure time for qemu
(b) reduce the level of thread-local storage in dependencies
(particularly ceph's stringify)
(c) increase the stack size specified in xenstore's xs.c
Is there is any precedent/policy with regards to expected TLS and/or
stack usage for dependencies? Is the best course of action (b)? Or
perhaps reconsider the default size for (c)?
Thoughts? :)
^ permalink raw reply [flat|nested] 5+ messages in thread* [Qemu-devel] stack size limit issues with xen + qemu + rbd
@ 2016-09-16 20:55 ` Chris Patterson
0 siblings, 0 replies; 5+ messages in thread
From: Chris Patterson @ 2016-09-16 20:55 UTC (permalink / raw)
To: xen-devel, qemu-devel, ceph-devel
I have spent some time investigating a case where qemu is failing to
register xenstore watches for a PV guest once I enable vfb (and
thereby triggering the creation of a qemu instance).
The qemu logs show something along the lines of:
xen be core: xen be core: xen be: watching backend path
(backend/console/3) failed
xen be: watching backend path (backend/console/3) failed
xen be core: xen be core: xen be: watching backend path (backend/vkbd/3) failed
xen be: watching backend path (backend/vkbd/3) failed
xen be core: xen be core: xen be: watching backend path (backend/qdisk/3) failed
xen be: watching backend path (backend/qdisk/3) failed
xen be core: xen be core: xen be: watching backend path (backend/qusb/3) failed
xen be: watching backend path (backend/qusb/3) failed
xen be core: xen be core: xen be: watching backend path (backend/vfb/3) failed
xen be: watching backend path (backend/vfb/3) failed
xen be core: xen be core: xen be: watching backend path (backend/qnic/3) failed
xen be: watching backend path (backend/qnic/3) failed
I have tested qemu master, qemu-xen in the master xen tree, as well as
a few tags all with the same issue.
I came across a similar issue reported by Juergen Gross:
https://lists.nongnu.org/archive/html/qemu-devel/2016-07/msg03341.html
Sure enough, the thread stack size was the culprit. I had been
testing with qemu with the associated fix "vnc-tight: fix regression
with libxenstore" as it is in master, so that wasn't it...
I did some basic analysis of the qemu binary and the libraries it is pulling in:
for lib in $(ldd /usr/local/bin/qemu-system-i386 | grep -o '/.* '); do
echo "lib=$lib"; readelf -S "$lib" | grep -e tbss -e tdata -A1 ; done
The largest consumers were:
lib=/usr/lib/x86_64-linux-gnu/librbd.so.1
[17] .tbss NOBITS 000000000088fed0 0068fed0
0000000000001820 0000000000000000 WAT 0 0 8
lib=/usr/lib/x86_64-linux-gnu/librados.so.2
[17] .tbss NOBITS 0000000000718600 00518600
0000000000000aa0 0000000000000000 WAT 0 0 8
IIUC, librbd & librados are using nearly 9k of the 16k alone (I am
assuming this thread-local storage must be consumed as part of the
thread's stack)?
I narrowed that down to Ceph's usage of __thread in stringify() in
src/include/stringify.h.
To make things functional, the options were either to:
(a) disable rbd at configure time for qemu
(b) reduce the level of thread-local storage in dependencies
(particularly ceph's stringify)
(c) increase the stack size specified in xenstore's xs.c
Is there is any precedent/policy with regards to expected TLS and/or
stack usage for dependencies? Is the best course of action (b)? Or
perhaps reconsider the default size for (c)?
Thoughts? :)
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [Xen-devel] stack size limit issues with xen + qemu + rbd
2016-09-16 20:55 ` [Qemu-devel] " Chris Patterson
@ 2016-09-19 19:50 ` Konrad Rzeszutek Wilk
-1 siblings, 0 replies; 5+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-09-19 19:50 UTC (permalink / raw)
To: Chris Patterson; +Cc: xen-devel, qemu-devel, ceph-devel
On Fri, Sep 16, 2016 at 04:55:17PM -0400, Chris Patterson wrote:
> I have spent some time investigating a case where qemu is failing to
> register xenstore watches for a PV guest once I enable vfb (and
> thereby triggering the creation of a qemu instance).
>
> The qemu logs show something along the lines of:
> xen be core: xen be core: xen be: watching backend path
> (backend/console/3) failed
> xen be: watching backend path (backend/console/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/vkbd/3) failed
> xen be: watching backend path (backend/vkbd/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/qdisk/3) failed
> xen be: watching backend path (backend/qdisk/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/qusb/3) failed
> xen be: watching backend path (backend/qusb/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/vfb/3) failed
> xen be: watching backend path (backend/vfb/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/qnic/3) failed
> xen be: watching backend path (backend/qnic/3) failed
>
> I have tested qemu master, qemu-xen in the master xen tree, as well as
> a few tags all with the same issue.
>
> I came across a similar issue reported by Juergen Gross:
> https://lists.nongnu.org/archive/html/qemu-devel/2016-07/msg03341.html
>
> Sure enough, the thread stack size was the culprit. I had been
> testing with qemu with the associated fix "vnc-tight: fix regression
> with libxenstore" as it is in master, so that wasn't it...
>
> I did some basic analysis of the qemu binary and the libraries it is pulling in:
>
> for lib in $(ldd /usr/local/bin/qemu-system-i386 | grep -o '/.* '); do
> echo "lib=$lib"; readelf -S "$lib" | grep -e tbss -e tdata -A1 ; done
>
> The largest consumers were:
> lib=/usr/lib/x86_64-linux-gnu/librbd.so.1
> [17] .tbss NOBITS 000000000088fed0 0068fed0
> 0000000000001820 0000000000000000 WAT 0 0 8
> lib=/usr/lib/x86_64-linux-gnu/librados.so.2
> [17] .tbss NOBITS 0000000000718600 00518600
> 0000000000000aa0 0000000000000000 WAT 0 0 8
>
> IIUC, librbd & librados are using nearly 9k of the 16k alone (I am
> assuming this thread-local storage must be consumed as part of the
> thread's stack)?
>
> I narrowed that down to Ceph's usage of __thread in stringify() in
> src/include/stringify.h.
>
> To make things functional, the options were either to:
> (a) disable rbd at configure time for qemu
> (b) reduce the level of thread-local storage in dependencies
> (particularly ceph's stringify)
> (c) increase the stack size specified in xenstore's xs.c
I would say c) for now and focus on b) long-term until c) can be
reverted?
>
> Is there is any precedent/policy with regards to expected TLS and/or
> stack usage for dependencies? Is the best course of action (b)? Or
No precendent/policy that I know of..
> perhaps reconsider the default size for (c)?
>
> Thoughts? :)
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] [Xen-devel] stack size limit issues with xen + qemu + rbd
@ 2016-09-19 19:50 ` Konrad Rzeszutek Wilk
0 siblings, 0 replies; 5+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-09-19 19:50 UTC (permalink / raw)
To: Chris Patterson; +Cc: xen-devel, qemu-devel, ceph-devel
On Fri, Sep 16, 2016 at 04:55:17PM -0400, Chris Patterson wrote:
> I have spent some time investigating a case where qemu is failing to
> register xenstore watches for a PV guest once I enable vfb (and
> thereby triggering the creation of a qemu instance).
>
> The qemu logs show something along the lines of:
> xen be core: xen be core: xen be: watching backend path
> (backend/console/3) failed
> xen be: watching backend path (backend/console/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/vkbd/3) failed
> xen be: watching backend path (backend/vkbd/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/qdisk/3) failed
> xen be: watching backend path (backend/qdisk/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/qusb/3) failed
> xen be: watching backend path (backend/qusb/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/vfb/3) failed
> xen be: watching backend path (backend/vfb/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/qnic/3) failed
> xen be: watching backend path (backend/qnic/3) failed
>
> I have tested qemu master, qemu-xen in the master xen tree, as well as
> a few tags all with the same issue.
>
> I came across a similar issue reported by Juergen Gross:
> https://lists.nongnu.org/archive/html/qemu-devel/2016-07/msg03341.html
>
> Sure enough, the thread stack size was the culprit. I had been
> testing with qemu with the associated fix "vnc-tight: fix regression
> with libxenstore" as it is in master, so that wasn't it...
>
> I did some basic analysis of the qemu binary and the libraries it is pulling in:
>
> for lib in $(ldd /usr/local/bin/qemu-system-i386 | grep -o '/.* '); do
> echo "lib=$lib"; readelf -S "$lib" | grep -e tbss -e tdata -A1 ; done
>
> The largest consumers were:
> lib=/usr/lib/x86_64-linux-gnu/librbd.so.1
> [17] .tbss NOBITS 000000000088fed0 0068fed0
> 0000000000001820 0000000000000000 WAT 0 0 8
> lib=/usr/lib/x86_64-linux-gnu/librados.so.2
> [17] .tbss NOBITS 0000000000718600 00518600
> 0000000000000aa0 0000000000000000 WAT 0 0 8
>
> IIUC, librbd & librados are using nearly 9k of the 16k alone (I am
> assuming this thread-local storage must be consumed as part of the
> thread's stack)?
>
> I narrowed that down to Ceph's usage of __thread in stringify() in
> src/include/stringify.h.
>
> To make things functional, the options were either to:
> (a) disable rbd at configure time for qemu
> (b) reduce the level of thread-local storage in dependencies
> (particularly ceph's stringify)
> (c) increase the stack size specified in xenstore's xs.c
I would say c) for now and focus on b) long-term until c) can be
reverted?
>
> Is there is any precedent/policy with regards to expected TLS and/or
> stack usage for dependencies? Is the best course of action (b)? Or
No precendent/policy that I know of..
> perhaps reconsider the default size for (c)?
>
> Thoughts? :)
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: stack size limit issues with xen + qemu + rbd
2016-09-16 20:55 ` [Qemu-devel] " Chris Patterson
(?)
(?)
@ 2016-09-19 19:50 ` Konrad Rzeszutek Wilk
-1 siblings, 0 replies; 5+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-09-19 19:50 UTC (permalink / raw)
To: Chris Patterson; +Cc: xen-devel, ceph-devel, qemu-devel
On Fri, Sep 16, 2016 at 04:55:17PM -0400, Chris Patterson wrote:
> I have spent some time investigating a case where qemu is failing to
> register xenstore watches for a PV guest once I enable vfb (and
> thereby triggering the creation of a qemu instance).
>
> The qemu logs show something along the lines of:
> xen be core: xen be core: xen be: watching backend path
> (backend/console/3) failed
> xen be: watching backend path (backend/console/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/vkbd/3) failed
> xen be: watching backend path (backend/vkbd/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/qdisk/3) failed
> xen be: watching backend path (backend/qdisk/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/qusb/3) failed
> xen be: watching backend path (backend/qusb/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/vfb/3) failed
> xen be: watching backend path (backend/vfb/3) failed
> xen be core: xen be core: xen be: watching backend path (backend/qnic/3) failed
> xen be: watching backend path (backend/qnic/3) failed
>
> I have tested qemu master, qemu-xen in the master xen tree, as well as
> a few tags all with the same issue.
>
> I came across a similar issue reported by Juergen Gross:
> https://lists.nongnu.org/archive/html/qemu-devel/2016-07/msg03341.html
>
> Sure enough, the thread stack size was the culprit. I had been
> testing with qemu with the associated fix "vnc-tight: fix regression
> with libxenstore" as it is in master, so that wasn't it...
>
> I did some basic analysis of the qemu binary and the libraries it is pulling in:
>
> for lib in $(ldd /usr/local/bin/qemu-system-i386 | grep -o '/.* '); do
> echo "lib=$lib"; readelf -S "$lib" | grep -e tbss -e tdata -A1 ; done
>
> The largest consumers were:
> lib=/usr/lib/x86_64-linux-gnu/librbd.so.1
> [17] .tbss NOBITS 000000000088fed0 0068fed0
> 0000000000001820 0000000000000000 WAT 0 0 8
> lib=/usr/lib/x86_64-linux-gnu/librados.so.2
> [17] .tbss NOBITS 0000000000718600 00518600
> 0000000000000aa0 0000000000000000 WAT 0 0 8
>
> IIUC, librbd & librados are using nearly 9k of the 16k alone (I am
> assuming this thread-local storage must be consumed as part of the
> thread's stack)?
>
> I narrowed that down to Ceph's usage of __thread in stringify() in
> src/include/stringify.h.
>
> To make things functional, the options were either to:
> (a) disable rbd at configure time for qemu
> (b) reduce the level of thread-local storage in dependencies
> (particularly ceph's stringify)
> (c) increase the stack size specified in xenstore's xs.c
I would say c) for now and focus on b) long-term until c) can be
reverted?
>
> Is there is any precedent/policy with regards to expected TLS and/or
> stack usage for dependencies? Is the best course of action (b)? Or
No precendent/policy that I know of..
> perhaps reconsider the default size for (c)?
>
> Thoughts? :)
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-09-19 19:51 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-09-16 20:55 stack size limit issues with xen + qemu + rbd Chris Patterson
2016-09-16 20:55 ` [Qemu-devel] " Chris Patterson
2016-09-19 19:50 ` [Xen-devel] " Konrad Rzeszutek Wilk
2016-09-19 19:50 ` [Qemu-devel] " Konrad Rzeszutek Wilk
2016-09-19 19:50 ` Konrad Rzeszutek Wilk
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.