[Qemu-devel] slirp-related crash

[Qemu-devel] slirp-related crash

[Michael S. Tsirkin]

It seems somewhat easy to crash qemu with slirp if we queue multiple packets.
I didn't investigate further yet so I don't know if this
is a regression. Anyone knowledgeable about slirp wants to take a look?

/home/mst/qemu-test/bin/qemu-system-x86_64  -enable-kvm -m 1G -drive
file=/home/mst/rhel6.qcow2 -netdev user,id=bar -net
nic,netdev=bar,model=e1000,macaddr=52:54:00:12:34:57  -redir
tcp:8022::22  -vnc :1 -monitor stdio

While guest is booting, quickly do this

ssh localhost -p 8022
CTRL-C
ssh localhost -p 8022
CTRL-C
ssh localhost -p 8022
CTRL-C
ssh localhost -p 8022
CTRL-C

When guest triest to bring up link,
qemu crashes:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7e4f8a7 in slirp_insque (a=0x0, b=0x7ffff91681f0) at
slirp/misc.c:27
27              element->qh_link = head->qh_link;
(gdb) where
#0  0x00007ffff7e4f8a7 in slirp_insque (a=0x0, b=0x7ffff91681f0) at
slirp/misc.c:27
#1  0x00007ffff7e4ddd8 in if_start (slirp=0x7ffff8b0e4f0) at
slirp/if.c:194
#2  0x00007ffff7e51290 in slirp_select_poll (readfds=0x7fffffffdfe0,
writefds=
    0x7fffffffdf60, xfds=0x7fffffffdee0, select_error=0) at
slirp/slirp.c:588
#3  0x00007ffff7e114c3 in main_loop_wait (nonblocking=<value optimized
out>)
    at main-loop.c:466
#4  0x00007ffff7e09ed4 in main_loop (argc=<value optimized out>, 
    argv=<value optimized out>, envp=<value optimized out>)
    at /home/mst/scm/qemu/vl.c:1482
#5  main (argc=<value optimized out>, argv=<value optimized out>, 
    envp=<value optimized out>) at /home/mst/scm/qemu/vl.c:3525
(gdb) p element
$1 = (struct quehead *) 0x0


-- 
MST

Re: [Qemu-devel] slirp-related crash

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1991 bytes --]

On 2012-02-12 19:34, Michael S. Tsirkin wrote:
> It seems somewhat easy to crash qemu with slirp if we queue multiple packets.
> I didn't investigate further yet so I don't know if this
> is a regression. Anyone knowledgeable about slirp wants to take a look?
> 
> /home/mst/qemu-test/bin/qemu-system-x86_64  -enable-kvm -m 1G -drive
> file=/home/mst/rhel6.qcow2 -netdev user,id=bar -net
> nic,netdev=bar,model=e1000,macaddr=52:54:00:12:34:57  -redir
> tcp:8022::22  -vnc :1 -monitor stdio
> 
> While guest is booting, quickly do this
> 
> ssh localhost -p 8022
> CTRL-C
> ssh localhost -p 8022
> CTRL-C
> ssh localhost -p 8022
> CTRL-C
> ssh localhost -p 8022
> CTRL-C

Confirmed. A single canceled connection prior the interface setup is
enough. Possibly something is not properly removed / cleaned up here.
Will see if I find some time to debug, can't promise.

Jan

> 
> When guest triest to bring up link,
> qemu crashes:
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff7e4f8a7 in slirp_insque (a=0x0, b=0x7ffff91681f0) at
> slirp/misc.c:27
> 27              element->qh_link = head->qh_link;
> (gdb) where
> #0  0x00007ffff7e4f8a7 in slirp_insque (a=0x0, b=0x7ffff91681f0) at
> slirp/misc.c:27
> #1  0x00007ffff7e4ddd8 in if_start (slirp=0x7ffff8b0e4f0) at
> slirp/if.c:194
> #2  0x00007ffff7e51290 in slirp_select_poll (readfds=0x7fffffffdfe0,
> writefds=
>     0x7fffffffdf60, xfds=0x7fffffffdee0, select_error=0) at
> slirp/slirp.c:588
> #3  0x00007ffff7e114c3 in main_loop_wait (nonblocking=<value optimized
> out>)
>     at main-loop.c:466
> #4  0x00007ffff7e09ed4 in main_loop (argc=<value optimized out>, 
>     argv=<value optimized out>, envp=<value optimized out>)
>     at /home/mst/scm/qemu/vl.c:1482
> #5  main (argc=<value optimized out>, argv=<value optimized out>, 
>     envp=<value optimized out>) at /home/mst/scm/qemu/vl.c:3525
> (gdb) p element
> $1 = (struct quehead *) 0x0
> 
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] slirp-related crash

[Zhi Yong Wu]

On Mon, Feb 13, 2012 at 4:24 AM, Jan Kiszka <jan.kiszka@web.de> wrote:
> On 2012-02-12 19:34, Michael S. Tsirkin wrote:
>> It seems somewhat easy to crash qemu with slirp if we queue multiple packets.
>> I didn't investigate further yet so I don't know if this
>> is a regression. Anyone knowledgeable about slirp wants to take a look?
>>
>> /home/mst/qemu-test/bin/qemu-system-x86_64  -enable-kvm -m 1G -drive
>> file=/home/mst/rhel6.qcow2 -netdev user,id=bar -net
>> nic,netdev=bar,model=e1000,macaddr=52:54:00:12:34:57  -redir
>> tcp:8022::22  -vnc :1 -monitor stdio
>>
>> While guest is booting, quickly do this
>>
>> ssh localhost -p 8022
>> CTRL-C
>> ssh localhost -p 8022
>> CTRL-C
>> ssh localhost -p 8022
>> CTRL-C
>> ssh localhost -p 8022
>> CTRL-C
>
> Confirmed. A single canceled connection prior the interface setup is
> enough. Possibly something is not properly removed / cleaned up here.
> Will see if I find some time to debug, can't promise.
Interesting thing, pls give me some time, and i am trying to debug this issue.

>
> Jan
>
>>
>> When guest triest to bring up link,
>> qemu crashes:
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x00007ffff7e4f8a7 in slirp_insque (a=0x0, b=0x7ffff91681f0) at
>> slirp/misc.c:27
>> 27              element->qh_link = head->qh_link;
>> (gdb) where
>> #0  0x00007ffff7e4f8a7 in slirp_insque (a=0x0, b=0x7ffff91681f0) at
>> slirp/misc.c:27
>> #1  0x00007ffff7e4ddd8 in if_start (slirp=0x7ffff8b0e4f0) at
>> slirp/if.c:194
>> #2  0x00007ffff7e51290 in slirp_select_poll (readfds=0x7fffffffdfe0,
>> writefds=
>>     0x7fffffffdf60, xfds=0x7fffffffdee0, select_error=0) at
>> slirp/slirp.c:588
>> #3  0x00007ffff7e114c3 in main_loop_wait (nonblocking=<value optimized
>> out>)
>>     at main-loop.c:466
>> #4  0x00007ffff7e09ed4 in main_loop (argc=<value optimized out>,
>>     argv=<value optimized out>, envp=<value optimized out>)
>>     at /home/mst/scm/qemu/vl.c:1482
>> #5  main (argc=<value optimized out>, argv=<value optimized out>,
>>     envp=<value optimized out>) at /home/mst/scm/qemu/vl.c:3525
>> (gdb) p element
>> $1 = (struct quehead *) 0x0
>>
>>
>
>



-- 
Regards,

Zhi Yong Wu

Re: [Qemu-devel] slirp-related crash

[Jan Kiszka]

On 2012-02-13 16:27, Zhi Yong Wu wrote:
> On Mon, Feb 13, 2012 at 4:24 AM, Jan Kiszka <jan.kiszka@web.de> wrote:
>> On 2012-02-12 19:34, Michael S. Tsirkin wrote:
>>> It seems somewhat easy to crash qemu with slirp if we queue multiple packets.
>>> I didn't investigate further yet so I don't know if this
>>> is a regression. Anyone knowledgeable about slirp wants to take a look?
>>>
>>> /home/mst/qemu-test/bin/qemu-system-x86_64  -enable-kvm -m 1G -drive
>>> file=/home/mst/rhel6.qcow2 -netdev user,id=bar -net
>>> nic,netdev=bar,model=e1000,macaddr=52:54:00:12:34:57  -redir
>>> tcp:8022::22  -vnc :1 -monitor stdio
>>>
>>> While guest is booting, quickly do this
>>>
>>> ssh localhost -p 8022
>>> CTRL-C
>>> ssh localhost -p 8022
>>> CTRL-C
>>> ssh localhost -p 8022
>>> CTRL-C
>>> ssh localhost -p 8022
>>> CTRL-C
>>
>> Confirmed. A single canceled connection prior the interface setup is
>> enough. Possibly something is not properly removed / cleaned up here.
>> Will see if I find some time to debug, can't promise.
> Interesting thing, pls give me some time, and i am trying to debug this issue.

I had a look today, but haven't found a fix yet. The problem is related
to our requeuing of packets if the target MAC is not yet known.
Something goes terribly wrong once it gets resolved (mbuf use after
release?). Maybe it was always wrong and the requeuing just surfaced the
bug, dunno.

After starring at the code for a while, I got the bad feeling of
"unfixable with reasonable effort". The queuing code is horrible (well,
like most of slirp), and the requeuing just made it worse. But maybe I'm
just missing some trick now - yet another one that would make the code
even more unreadable...

I'm inclined to suggest a slirp rewrite (base support, not all features
at once) as a GSOC project. QEMU really deserves something better.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] slirp-related crash

[Jan Kiszka]

On 2012-02-13 20:35, Jan Kiszka wrote:
> On 2012-02-13 16:27, Zhi Yong Wu wrote:
>> On Mon, Feb 13, 2012 at 4:24 AM, Jan Kiszka <jan.kiszka@web.de> wrote:
>>> On 2012-02-12 19:34, Michael S. Tsirkin wrote:
>>>> It seems somewhat easy to crash qemu with slirp if we queue multiple packets.
>>>> I didn't investigate further yet so I don't know if this
>>>> is a regression. Anyone knowledgeable about slirp wants to take a look?
>>>>
>>>> /home/mst/qemu-test/bin/qemu-system-x86_64  -enable-kvm -m 1G -drive
>>>> file=/home/mst/rhel6.qcow2 -netdev user,id=bar -net
>>>> nic,netdev=bar,model=e1000,macaddr=52:54:00:12:34:57  -redir
>>>> tcp:8022::22  -vnc :1 -monitor stdio
>>>>
>>>> While guest is booting, quickly do this
>>>>
>>>> ssh localhost -p 8022
>>>> CTRL-C
>>>> ssh localhost -p 8022
>>>> CTRL-C
>>>> ssh localhost -p 8022
>>>> CTRL-C
>>>> ssh localhost -p 8022
>>>> CTRL-C
>>>
>>> Confirmed. A single canceled connection prior the interface setup is
>>> enough. Possibly something is not properly removed / cleaned up here.
>>> Will see if I find some time to debug, can't promise.
>> Interesting thing, pls give me some time, and i am trying to debug this issue.
> 
> I had a look today, but haven't found a fix yet. The problem is related
> to our requeuing of packets if the target MAC is not yet known.
> Something goes terribly wrong once it gets resolved (mbuf use after
> release?). Maybe it was always wrong and the requeuing just surfaced the
> bug, dunno.
> 
> After starring at the code for a while, I got the bad feeling of
> "unfixable with reasonable effort". The queuing code is horrible (well,
> like most of slirp), and the requeuing just made it worse. But maybe I'm
> just missing some trick now - yet another one that would make the code
> even more unreadable...
> 
> I'm inclined to suggest a slirp rewrite (base support, not all features
> at once) as a GSOC project. QEMU really deserves something better.
> 

Despite my grumbling, I would welcome any patch that fixes the bug, of
course!

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] slirp-related crash

[Alex Bradbury]

On 13 February 2012 19:35, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> I'm inclined to suggest a slirp rewrite (base support, not all features
> at once) as a GSOC project. QEMU really deserves something better.

Would it be feasible to adopt uip from kvm tool instead?

https://github.com/penberg/linux-kvm/tree/master/tools/kvm/net/uip
http://www.linux-kvm.org/wiki/images/c/c5/2011-forum-native-linux-kvm-tool.pdf

Alex

Re: [Qemu-devel] slirp-related crash

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 701 bytes --]

On 2012-02-13 21:43, Alex Bradbury wrote:
> On 13 February 2012 19:35, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> I'm inclined to suggest a slirp rewrite (base support, not all features
>> at once) as a GSOC project. QEMU really deserves something better.
> 
> Would it be feasible to adopt uip from kvm tool instead?
> 
> https://github.com/penberg/linux-kvm/tree/master/tools/kvm/net/uip
> http://www.linux-kvm.org/wiki/images/c/c5/2011-forum-native-linux-kvm-tool.pdf

Surely a possible starting point. Or lwip, which also has v6. In any
case, there will be enough work to properly plug things into the QEMU
infrastructure and prepare them for the features slirp provides.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] slirp-related crash

[Stefan Hajnoczi]

On Mon, Feb 13, 2012 at 08:35:45PM +0100, Jan Kiszka wrote:
> On 2012-02-13 16:27, Zhi Yong Wu wrote:
> I'm inclined to suggest a slirp rewrite (base support, not all features
> at once) as a GSOC project. QEMU really deserves something better.

A talented student could pull it off, especially if integrating lwip or
uip instead of writing from scratch.  It would be important to have a
list of requirements and clear plan so that if they are unable to
complete it fully, we still have a basic but working implementation to
build on.

But it's a lot of work to make it feature-complete and comparable to
slirp.

Current feature set:

 * UDPv4 NAT
 * TCPv4 NAT
 * ICMP for virtual interfaces (host, DNS, SAMBA)
 * DNS integration
 * DHCP integration
 * TFTP integration
 * UDPv4 port forwarding
 * TCPv4 port forwarding
 * SAMBA integration

Stefan

Re: [Qemu-devel] slirp-related crash

[Jan Kiszka]

On 2012-02-14 09:22, Stefan Hajnoczi wrote:
> On Mon, Feb 13, 2012 at 08:35:45PM +0100, Jan Kiszka wrote:
>> On 2012-02-13 16:27, Zhi Yong Wu wrote:
>> I'm inclined to suggest a slirp rewrite (base support, not all features
>> at once) as a GSOC project. QEMU really deserves something better.
> 
> A talented student could pull it off, especially if integrating lwip or
> uip instead of writing from scratch.  It would be important to have a
> list of requirements and clear plan so that if they are unable to
> complete it fully, we still have a basic but working implementation to
> build on.

Yep.

> 
> But it's a lot of work to make it feature-complete and comparable to
> slirp.
> 
> Current feature set:
> 
>  * UDPv4 NAT
>  * TCPv4 NAT
>  * ICMP for virtual interfaces (host, DNS, SAMBA)

...also for routed hosts (on Linux 3.0+)

>  * DNS integration
>  * DHCP integration
>  * TFTP integration

/me wonders if dnsmasq could be of some help. I don't think you can
attach it already to something else than a network interface. But maybe
it could be extended to run against a channel provided by some slirp 2.0.

>  * UDPv4 port forwarding
>  * TCPv4 port forwarding

Including forwarding from the guest to host-side chardev backends.

>  * SAMBA integration

And all this (almost) fully configurable.

Yes, it's a lot of stuff, much more than even a talented student could
complete in a single GSOC. I'll try to specify a reasonable, possibly
scalable task.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux