From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34266) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c3hO9-0005t7-2F for qemu-devel@nongnu.org; Mon, 07 Nov 2016 05:42:54 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1c3hO5-0003OJ-4R for qemu-devel@nongnu.org; Mon, 07 Nov 2016 05:42:53 -0500 Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]:35538) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1c3hO4-0003OC-Q2 for qemu-devel@nongnu.org; Mon, 07 Nov 2016 05:42:49 -0500 Received: by mail-wm0-x242.google.com with SMTP id a20so4902612wme.2 for ; Mon, 07 Nov 2016 02:42:48 -0800 (PST) Date: Mon, 7 Nov 2016 10:42:45 +0000 From: Stefan Hajnoczi Message-ID: <20161107104245.GC5036@stefanha-x1.localdomain> References: <95e79bc8-4547-b3b1-65b7-f641eb0c92f7@pobox.com> <20161104111419.GG9817@stefanha-x1.localdomain> <20161106180401.GE27308@var.home> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="GPJrCs/72TxItFYR" Content-Disposition: inline In-Reply-To: Subject: Re: [Qemu-devel] Crashing in tcp_close List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Brian Candler Cc: Samuel Thibault , qemu-devel@nongnu.org, Jan Kiszka --GPJrCs/72TxItFYR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Nov 07, 2016 at 08:42:17AM +0000, Brian Candler wrote: > On 06/11/2016 18:04, Samuel Thibault wrote: > > Brian, could you run it with > >=20 > > export MALLOC_CHECK_=3D2 > >=20 > > and also this could be useful: > >=20 > > export MALLOC_PERTURB_=3D1234 > >=20 > > Also, to rule out the double-free scenario, and try to catch a buffer > > overflow coming from the socket structure itself, I have attached a > > patch which adds some debugging. >=20 > Thanks. I've added the patch, and re-run the stress test. >=20 > (Aside: since last post I've replaced the Mac Mini with Intel NUCi6KYK so > it's not *exactly* the same environment, although both machines are > quad-core i7) >=20 > Unfortunately it doesn't crash every time. Here are the first two crashes > I've managed to obtain, and they don't seem to by anything to do with > tcp_close, but I have pasted them below. I have kept the coredumps if > there's anything more useful I can extract from them. >=20 > The full command line is something like this (taken from a later run): >=20 > /usr/local/bin/qemu-system-x86_64 -m 4G -machine type=3Dpc,accel=3Dkvm -n= etdev > user,id=3Duser.0,hostfwd=3Dtcp::2305-:22 -device virtio-scsi-pci,id=3Dscs= i0 > -device scsi-hd,bus=3Dscsi0.0,drive=3Ddrive0 -device virtio-net,netdev=3D= user.0 > -name vtp-nmm-201611070837.qcow2 -drive if=3Dnone,file=3Doutput-qemu-vtp-= nmm/vtp-nmm-201611070837.qcow2,id=3Ddrive0,cache=3Dwriteback,discard=3Dunma= p,format=3Dqcow2 > -boot c -vnc [::]:24 >=20 > The following crashes occurred when running with a single vcpu. Normally I > have been running with -smp 8,sockets=3D1,cores=3D4,threads=3D2 as it see= ms to > crash less with those settings; however I'm trying it again like that in a > loop to see if I can get a crash. Let's try to isolate the cause of this crash: Are you able to switch -netdev user to -netdev tap so we can rule out the slirp user network stack as the source of memory corruption? Alternatively could you re-run with virtio-blk instead of virtio-scsi to see if that eliminates crashes? The core dumps are likely to contain more clues. If you are comfortable with gdb and debugging C code you could dump the memory surround where the junk value (mr) was loaded from. Perhaps there is a hint about who zeroed the memory. In the first core dump you could start with: (gdb) up 6 # go to the dma_blk_unmap() stack frame (gdb) p *(DMAAIOCB *)0x560909ceca90 (gdb) p *((DMAAIOCB *)0x560909ceca90).sg > Regards, >=20 > Brian. >=20 > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > Core was generated by `/usr/local/bin/qemu-system-x86_64 -m 4G -name > vtp-nmm-201611062024.qcow2 -machi'. > Program terminated with signal SIGABRT, Aborted. > #0 0x00007f366c4ce428 in __GI_raise (sig=3Dsig@entry=3D6) at > ../sysdeps/unix/sysv/linux/raise.c:54 > 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. > [Current thread is 1 (Thread 0x7f366deeea80 (LWP 9030))] > (gdb) bt > #0 0x00007f366c4ce428 in __GI_raise (sig=3Dsig@entry=3D6) at > ../sysdeps/unix/sysv/linux/raise.c:54 > #1 0x00007f366c4d002a in __GI_abort () at abort.c:89 > #2 0x00007f366c4c6bd7 in __assert_fail_base (fmt=3D, > assertion=3Dassertion@entry=3D0x560907875cd5 "mr !=3D NULL", > file=3Dfile@entry=3D0x560907857884 "/home/nsrc/qemu-2.7.0/exec.c", > line=3Dline@entry=3D2967, > function=3Dfunction@entry=3D0x560907857f00 <__PRETTY_FUNCTION__.42881> > "address_space_unmap") > at assert.c:92 > #3 0x00007f366c4c6c82 in __GI___assert_fail ( > assertion=3Dassertion@entry=3D0x560907875cd5 "mr !=3D NULL", > file=3Dfile@entry=3D0x560907857884 "/home/nsrc/qemu-2.7.0/exec.c", > line=3Dline@entry=3D2967, > function=3Dfunction@entry=3D0x560907857f00 <__PRETTY_FUNCTION__.42881> > "address_space_unmap") > at assert.c:101 > #4 0x000056090749dffe in address_space_unmap (as=3D, > buffer=3D, > len=3D, is_write=3D1, access_len=3D8192) at > /home/nsrc/qemu-2.7.0/exec.c:2967 > #5 0x00005609075af586 in dma_memory_unmap (access_len=3D, > dir=3D, > len=3D, buffer=3D, as=3D) > at /home/nsrc/qemu-2.7.0/include/sysemu/dma.h:144 > #6 dma_blk_unmap (dbs=3Ddbs@entry=3D0x560909ceca90) at > /home/nsrc/qemu-2.7.0/dma-helpers.c:102 > #7 0x00005609075af766 in dma_complete (ret=3D0, dbs=3D0x560909ceca90) > at /home/nsrc/qemu-2.7.0/dma-helpers.c:113 > #8 dma_blk_cb (opaque=3D0x560909ceca90, ret=3D0) at > /home/nsrc/qemu-2.7.0/dma-helpers.c:137 > #9 0x000056090775d25a in blk_aio_complete (acb=3D0x56090909aba0) > at /home/nsrc/qemu-2.7.0/block/block-backend.c:923 > #10 0x00005609077ccaea in coroutine_trampoline (i0=3D, > i1=3D) > at /home/nsrc/qemu-2.7.0/util/coroutine-ucontext.c:78 > #11 0x00007f366c4e35d0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #12 0x00007fffd8d31f20 in ?? () > #13 0x2d2d2d2d2d2d2d2d in ?? () > #14 0x00000000000000d0 in ?? () > #15 0x0000000000000000 in ?? () > (gdb) >=20 >=20 > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > Core was generated by `/usr/local/bin/qemu-system-x86_64 -boot c -vnc > [::]:78 -name vtp-nmm-2016110621'. > Program terminated with signal SIGABRT, Aborted. > #0 0x00007fc226c98428 in __GI_raise (sig=3Dsig@entry=3D6) at > ../sysdeps/unix/sysv/linux/raise.c:54 > 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. > [Current thread is 1 (Thread 0x7fc2286b8a80 (LWP 10267))] > (gdb) bt > #0 0x00007fc226c98428 in __GI_raise (sig=3Dsig@entry=3D6) at > ../sysdeps/unix/sysv/linux/raise.c:54 > #1 0x00007fc226c9a02a in __GI_abort () at abort.c:89 > #2 0x00007fc226c90bd7 in __assert_fail_base (fmt=3D, > assertion=3Dassertion@entry=3D0x561867557cd5 "mr !=3D NULL", > file=3Dfile@entry=3D0x561867539884 "/home/nsrc/qemu-2.7.0/exec.c", > line=3Dline@entry=3D2967, > function=3Dfunction@entry=3D0x561867539f00 <__PRETTY_FUNCTION__.42881> > "address_space_unmap") > at assert.c:92 > #3 0x00007fc226c90c82 in __GI___assert_fail ( > assertion=3Dassertion@entry=3D0x561867557cd5 "mr !=3D NULL", > file=3Dfile@entry=3D0x561867539884 "/home/nsrc/qemu-2.7.0/exec.c", > line=3Dline@entry=3D2967, > function=3Dfunction@entry=3D0x561867539f00 <__PRETTY_FUNCTION__.42881> > "address_space_unmap") > at assert.c:101 > #4 0x000056186717fffe in address_space_unmap (as=3D, > buffer=3D, > len=3D, is_write=3D1, access_len=3D4096) at > /home/nsrc/qemu-2.7.0/exec.c:2967 > #5 0x0000561867202beb in virtqueue_unmap_sg > (elem=3Delem@entry=3D0x5618694fc610, len=3Dlen@entry=3D32876, > vq=3D0x5618695a8500) at /home/nsrc/qemu-2.7.0/hw/virtio/virtio.c:254 > #6 0x0000561867203422 in virtqueue_fill (vq=3Dvq@entry=3D0x5618695a8500, > elem=3Delem@entry=3D0x5618694fc610, len=3D32876, idx=3Didx@entry=3D0) > at /home/nsrc/qemu-2.7.0/hw/virtio/virtio.c:282 > #7 0x00005618672035db in virtqueue_push (vq=3Dvq@entry=3D0x5618695a8500, > elem=3Delem@entry=3D0x5618694fc610, len=3D) > at /home/nsrc/qemu-2.7.0/hw/virtio/virtio.c:308 > #8 0x00005618671f0885 in virtio_scsi_complete_req (req=3D0x5618694fc610) > at /home/nsrc/qemu-2.7.0/hw/scsi/virtio-scsi.c:70 > #9 0x00005618671f09e6 in virtio_scsi_complete_cmd_req (req=3D0x5618694fc= 610) > at /home/nsrc/qemu-2.7.0/hw/scsi/virtio-scsi.c:443 > #10 virtio_scsi_command_complete (r=3D, status=3D0, resid= =3D0) > at /home/nsrc/qemu-2.7.0/hw/scsi/virtio-scsi.c:470 > #11 0x0000561867365c98 in scsi_req_complete (req=3D0x561868a72e40, > status=3D) > at /home/nsrc/qemu-2.7.0/hw/scsi/scsi-bus.c:1775 > #12 0x0000561867360210 in scsi_dma_complete_noio (r=3D0x561868a72e40, > ret=3D) > at /home/nsrc/qemu-2.7.0/hw/scsi/scsi-disk.c:278 > ---Type to continue, or q to quit--- > #13 0x0000561867291779 in dma_complete (ret=3D0, dbs=3D0x561868371050) > at /home/nsrc/qemu-2.7.0/dma-helpers.c:115 > #14 dma_blk_cb (opaque=3D0x561868371050, ret=3D0) at > /home/nsrc/qemu-2.7.0/dma-helpers.c:137 > #15 0x000056186743f25a in blk_aio_complete (acb=3D0x561869661b90) > at /home/nsrc/qemu-2.7.0/block/block-backend.c:923 > #16 0x00005618674aeaea in coroutine_trampoline (i0=3D, > i1=3D) > at /home/nsrc/qemu-2.7.0/util/coroutine-ucontext.c:78 > #17 0x00007fc226cad5d0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #18 0x00007ffd10673d70 in ?? () > #19 0x2d2d2d2d2d2d2d2d in ?? () > #20 0x00000000000000d4 in ?? () > #21 0x0000000000000000 in ?? () > (gdb) >=20 >=20 --GPJrCs/72TxItFYR Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJYIFqlAAoJEJykq7OBq3PINK0H/ibmG8nJuPUe134xsJTRN4/e 3vi4Kenv0OT61DNrNJHHRH4iGoSqBKEERtKzqi3qnbTnqvOZRgYgjtIhHjUtpmDW jivXDWyUNFHZzlxS9zUOoauaMR3zpOF0Q/2b2t9mPkWa3HGDanAHXIOAd+W7uFov BHyYolSJsjXCtz5/8m/4oIwkgJDZeYMZOEK3K2cuFXrr5McxTm7gTZoVtcOgkusc b/Ql9JkJQTCg4u1LfGIAGLij7MTYsS7IeS4ifplSmUAgTP+YqZs/KU/3fT2+IHOg JMAesUDvKoJgF8qRRetFWeF2jvETxsxrBHi1oieeoWFMYzhwCSdqDf4p2MTUpVo= =N3Y+ -----END PGP SIGNATURE----- --GPJrCs/72TxItFYR--