From: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
To: Yi Zhang <yizhan-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>,
linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org,
Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
Tariq Toukan <tariqt-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Subject: Re: mlx4_core 0000:07:00.0: swiotlb buffer is full and OOM observed during stress test on reset_controller
Date: Thu, 9 Mar 2017 10:46:41 +0200 [thread overview]
Message-ID: <20170309084641.GY14379@mtr-leonro.local> (raw)
In-Reply-To: <20170305081206.GI14379-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 8305 bytes --]
On Sun, Mar 05, 2017 at 10:12:06AM +0200, Leon Romanovsky wrote:
> On Fri, Mar 03, 2017 at 06:55:11AM -0500, Yi Zhang wrote:
> > Hi experts
> >
> > I reproduced this issue during stress test on reset_controller, could you help check it, thanks.
> >
> > Reproduce steps on initiator side:
> > num=0
> > while [ 1 ]
> > do
> > echo "-------------------------------$num"
> > echo 1 >/sys/block/nvme0n1/device/reset_controller || exit 1
> > ((num++))
> > done
> >
> > Here is the full log:
> > http://pastebin.com/mek9fb0b
> >
> > Target side log:
> > [ 326.411481] nvmet: creating controller 1061 for NQN nqn.2014-08.org.nvmexpress:NVMf:uuid:32413b2b-89cc-4939-b816-399ff293800d.
> > [ 326.516226] nvmet: adding queue 1 to ctrl 1061.
> > [ 326.516428] nvmet: adding queue 2 to ctrl 1061.
> > [ 326.516616] nvmet: adding queue 3 to ctrl 1061.
> > [ 326.5361ing queue 4 to ctrl 1061.
> > [ 326.556148] nvmet: adding queue 5 to ctrl 1061.
> > [ 326.556499] nvmet: adding queue 6 to ctrl 1061.
> > [ 326.556779] nvmet: adding queue 7 to ctrl 1061.
> > [ 326.557093] nvmet: adding queue 8 to ctrl 1061.
> > [ 326.576166] nvmet: adding queue 9 to ctrl 1061.
> > [ 326.576420] nvmet: adding queue 10 to ctrl 1061.
> > [ 326.576674] nvmet: adding queue 11 to ctrl 1061.
> > [ 326.576922] nvmet: adding queue 12 to ctrl 1061.
> > [ 326.577274] nvmet: adding queue 13 to ctrl 1061.
> > [ 326.577595] nvmet: adding queue 14 to ctrl 1061.
> > [ 326.596656] nvmet: adding queue 15 to ctrl 1061.
> > [ 326.596936] nvmet: adding queue 16 to ctrl 1061.
> > [ 326.662587] nvmet: creating controller 1062 for NQN nqn.2014-08.org.nvmexpress:NVMf:uuid:32413b2b-89cc-4939-b816-399ff293800d.
> > [ 326.686765] mlx4_core 0000:07:00.0: swiotlb buffer is full (sz: 532480 bytes)
> > [ 326.686766] swiotlb: coherent allocation failed for device 0000:07:00.0 size=532480
> > [ 326.686768] CPU: 6 PID: 3931 Comm: kworker/6:256 Not tainted 4.10.0 #2
> > [ 326.686768] Hardware name: HP ProLiant DL388p Gen8, BIOS P70 12/20/2013
> > [ 326.686775] Workqueue: ib_cm cm_work_handler [ib_cm]
> > [ 326.686776] Call Trace:
> > [ 326.686781] dump_stack+0x63/0x87
> > [ 326.686783]herent+0x14a/0x160
> > [ 326.686786] x86_swiotlb_alloc_coherent+0x43/0x50
> > [ 326.686795] mlx4_buf_direct_alloc.isra.4+0xb1/0x150 [mlx4_core]
> > [ 326.686798] mlx4_buf_alloc+0x172/0x1c0 [mlx4_core]
> > [ 326.686802] create_qp_common.isra.33+0x633/0x1010 [mlx4_ib]
> > [ 326.686805] ? mlx4_ib_create_qp+0xf7/0x450 [mlx4_ib]
> > [ 326.686807] mlx4_ib_create_qp+0x149/0x450 [mlx4_ib]
> > [ 326.686816] ib_create_qp+0x70/0x2b0 [ib_core]
> > [ 326.686819] rdma_create_qp+0x34/0xa0 [rdma_cm]
> > [ 326.686823] nvmet_rdma_alloc_queue+0x692/0x900 [nvmet_rdma]
> > [ 326.686824] ? nvmet_rdma_execute_command+0x100/0x100 [nvmet_rdma]
> > [ 326.686826] nvmet_rdma_cm_handler+0x1e6/0x708 [nvmet_rdma]
> > [ 326.686827] ? cma_acquire_dev+0x1e7/0x4b0 [rdma_cm]
> > [ 326.686829] ? cma_new_conn_id+0xb2/0x4b0 [rdma_cm]
> > [ 326.686830] ? cma_new_conn_id+0x153/0x4b0 [rdma_cm]
> > [ 326.686832] cma_req_handler+0x1f5/0x4c0 [rdma_cm]
> > [ 326.686834] cm_process_work+0x25/0x120 [ib_cm]
> > [ 326.686835] cm_req_handler+0x994/0xcd0 [ib_cm]
> > [ 326.686837] cm_work_handler+0x1ce/0x1753 [ib_cm]
> > [ 326.686839] process_one_work+0x165/0x410
> > [ 326.686840] worker_thread+0x137/0x4c0
> > [ 326.686841] kthread+0x101/0x140
> > [ 326.686842] ? rescuer_thread+0x3b0/0x3b0
> > [ 326.686843] ? kthread_park+0x90/0x90
> > [ 326.686845] ret_from_fork+0x2c/0x40
> > [ 326.691158] mlx4_core 0000:07:00.0: swiotlb buffer is full (sz: 532480 bytes)
> > [ 326.691158] swiotlb: coherent allocevice 0000:07:00.0 size=532480
> > [ 326.691160] CPU: 6 PID: 3931 Comm: kworker/6:256 Not tainted 4.10.0 #2
> > [ 326.691160] Hardware name: HP ProLiant DL388p Gen8, BIOS P70 12/20/2013
> > [ 326.691163] Workqueue: ib_cm cm_work_handler [ib_cm]
> > [ 326.691163] Call Trace:
> > [ 326.691165] dump_stack+0x63/0x87
> > [ 326.691167] swiotlb_alloc_coherent+0x14a/0x160
> > [ 326.691168] x86_swiotlb_alloc_coherent+0x43/0x50
> > [ 326.691173] mlx4_buf_direct_alloc.isra.4+0xb1/0x150 [mlx4_core]
> > [ 326.691176] mlx4_buf_alloc+0x172/0x1c0 [mlx4_core]
> > [ 326.691179] create_qp_common.isra.33+0x633/0x1010 [mlx4_ib]
> > [ 326.691181] mlx4_ib_create_qp+0x149/0x450 [mlx4_ib]
> > [ 326.691186] ib_create_qp+0x70/0x2b0 [ib_core]
> > [ 326.691188] rdma_create_qp+0x34/0xa0 [rdma_cm]
> > [ 326.691190] nvmet_rdma_alloc_queue+0x692/0x900 [nvmet_rdma]
> > [ 326.691191] ? nvmet_rdma_execute_command+0x100/0x100 [nvmet_rdma]
> > [ 326.691193] nvmet_rdma_cm_handler+0x1e6/0x708 [nvmet_rdma]
> > [ 326.691194] ? cma_acquire_dev+0x1e7/0x4b0 [rdma_cm]
> > [ 326.691196] ? cma_new_conn_id+0xb2/0x4b0 [rdma_cm]
> > [ 326.691197] ? cma_new_conn_id+0x153/0x4b0 [rdma_cm]
> > [ 326.691199] cma_req_handler+0x1f5/0x4c0 [rdma_cm]
> > [ 326.691201] cm_process_work+0x25/0x120 [ib_cm]
> > [ 326.691202] cm_req_handler+0x994/0xcd0 [ib_cm]
> > [ 326.691204] cm_work_handler+0x1ce/0x1753 [ib_cm]
> > [ 326.691205] process_one_work+0x165/0x410
> > [ 326.691206] worker_thread+0x137/0x4c0
> > [ 326.691207] kthread+0x101/0x140
> > [ 326.691209] ? rescuer_thread+0x3b0/0x3b0
> > [ 326.691209] ? kthread_park+0x90/0x90
> > [ 326.691211] ret_from_fork+0x2c/0x40
> > [ 326.695215] mlx4_core 0000:07:00.0: swiotlb buffer is full (sz: 532480 bytes)
> > [ 326.695216] swiotlb: coherent allocation failed for device 0000:07:00.0 size=532480
> > [ 326.695217] CPU: 6 PID: 3931 Comm: kworker/6:256 Not tainted 4.10.0 #2
> > [ 326.695217] Hardware name: HP ProLiant DL388p Gen8, BIOS P70 12/20/2013
> > [ 326.695219] Workqueue: ib_cm cm_work_handler [ib_cm]
> > [ 326.695220] Call Trace:
> > [ 326.695222] dump_stack+0x63/0x87
> > [ .695223] swiotlb_alloc_coherent+0x14a/0x160
> > [ 326.695224] x86_swiotlb_alloc_coherent+0x43/0x50
> > [ 326.695228] mlx4_buf_direct_alloc.isra.4+0xb1/0x150 [mlx4_core]
> > [ 326.695232] mlx4_buf_alloc+0x172/0x1c0 [mlx4_core]
> > [ 326.695234] create_qp_common.isra.33+0x633/0x1010 [mlx4_ib]
> > [ 326.695237] mlx4_ib_create_qp+0x149/0x450 [mlx4_ib]
> > [ 326.695241] ib_create_qp+0x70/0x2b0 [ib_core]
> > [ 326.695243] rdma_create_qp+0x34/0xa0 [rdma_cm]
> > [ 326.695245] nvmet_rdma_alloc_queue+0x692/0x900 [nvmet_rdma]
> > [ 326.695246] ? nvmet_rdma_execute_command+0x100/0x100 [nvmet_rdma]
> > [ 326.695247] nvmet_rdma_cm_handler+0x1e6/0x708 [nvmet_rdma]
> > [ 326.695249] ? cma_acquire_dev+0x1e7/0x4b0 [rdma_cm]
> > [ 326.695251] ? cma_new_conn_id+0xb2/0x4b0 [rdma_cm]
> > [ 326.695252] ? cma_new_conn_id+0x153/0x4b0 [rdma_cm]
> > [ 326.695254] cma_req_handler+0x1f5/0x4c0 [rdma_cm]
> > [ 326.695256] cm_process_work+0x25/0x120 [ib_cm]
> > [ 326.695257] cm_req_handler+0x994/0xcd0 [ib_cm]
> > [ 326.695259] cm_work_handler+0x1ce/0x1753 [ib_cm]
> > [ 326.695260] process_one_work+0x165/0x410
> > [ 326.695261] worker_thread+0x137/0x4c0
> > [ 326.695262] kthread+0x101/0x140
> > [ 326.695263] ? rescuer_thread+0x3b0/0x3b0
> > [ 326.695264] ? kthread_park+0x90/0x90
> > [ 3t_from_fork+0x2c/0x40
> >
> > Initiator side log:
> > [ 532.880043] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.2.3:1023
> > [ 533.002002] nvme nvme0: creating 16 I/O queues.
> > [ 533.446540] nvme nvme0: new ctrl: NQN "nvme-subsystem-name", addr 172.31.2.3:1023
> > [ 691.641201] nvme nvme0: rdma_resolve_addr wait failed (-110).
> > [ 691.672089] nvme nvme0: failed to initialize i/o queue: -110
> > [ 691.721031] nvme nvme0: Removing after reset failure
>
> + Christoph and Sagi.
+Tariq and Yishai.
How can we know from this log which memory order failed?
It can be one of two: memory leak (most probably) or/and fragmented
memory.
>
> >
> >
> > Best Regards,
> > Yi Zhang
> >
> >
> >
> > _______________________________________________
> > Linux-nvme mailing list
> > Linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> > http://lists.infradead.org/mailman/listinfo/linux-nvme
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
next prev parent reply other threads:[~2017-03-09 8:46 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <1908657724.31179983.1488539944957.JavaMail.zimbra@redhat.com>
[not found] ` <1908657724.31179983.1488539944957.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2017-03-03 11:55 ` mlx4_core 0000:07:00.0: swiotlb buffer is full and OOM observed during stress test on reset_controller Yi Zhang
[not found] ` <2013049462.31187009.1488542111040.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2017-03-05 8:12 ` Leon Romanovsky
[not found] ` <20170305081206.GI14379-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-03-08 15:48 ` Christoph Hellwig
[not found] ` <20170308154815.GB24437-jcswGhMUV9g@public.gmane.org>
2017-03-09 8:42 ` Leon Romanovsky
2017-03-09 8:46 ` Leon Romanovsky [this message]
[not found] ` <20170309084641.GY14379-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-03-09 10:33 ` Yi Zhang
2017-03-06 11:23 ` Sagi Grimberg
[not found] ` <95e045a8-ace0-6a9a-b9a9-555cb2670572-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-03-09 4:20 ` Yi Zhang
[not found] ` <d21c5571-78fd-7882-b4cc-c24f76f6ff47-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2017-03-10 16:52 ` Leon Romanovsky
[not found] ` <20170310165214.GC14379-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-03-12 18:16 ` Max Gurtovoy
[not found] ` <56e8ccd3-8116-89a1-2f65-eb61a91c5f84-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2017-03-14 13:35 ` Yi Zhang
[not found] ` <860db62d-ae93-d94c-e5fb-88e7b643f737-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2017-03-14 16:52 ` Max Gurtovoy
[not found] ` <0a825b18-df06-9a6d-38c9-402f4ee121f7-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2017-03-15 7:48 ` Yi Zhang
[not found] ` <7496c68a-15f3-d8cb-b17f-20f5a59a24d2-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2017-03-16 16:51 ` Sagi Grimberg
[not found] ` <31678a43-f76c-a921-e40c-470b0de1a86c-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-03-18 11:51 ` Yi Zhang
[not found] ` <1768681609.3995777.1489837916289.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2017-03-18 17:50 ` Sagi Grimberg
2017-03-19 7:01 ` Leon Romanovsky
[not found] ` <20170319070115.GP2079-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-05-18 17:01 ` Yi Zhang
[not found] ` <136275928.8307994.1495126919829.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2017-05-19 16:17 ` Yi Zhang
[not found] ` <358169046.8629042.1495210672801.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2017-06-04 15:49 ` Sagi Grimberg
[not found] ` <6bf26cbc-71e4-a030-628b-a2ee1d1de94b-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-06-15 8:45 ` Yi Zhang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170309084641.GY14379@mtr-leonro.local \
--to=leon-dgejt+ai2ygdnm+yrofe0a@public.gmane.org \
--cc=hch-jcswGhMUV9g@public.gmane.org \
--cc=linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org \
--cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org \
--cc=tariqt-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \
--cc=yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \
--cc=yizhan-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).