PROBLEM: call trace triggered in 5.1.1 in drivers/nvme/host/pci.c, 5.0.11 ok

[ming.lei@redhat.com (Ming Lei)]

On Tue, May 14, 2019@07:54:34AM -0600, Keith Busch wrote:
> On Tue, May 14, 2019@04:24:41PM +1000, Adam Carter wrote:
> > Ok i've rebooted into 5.1.1 to get the whole thing - see attached.
> > 
> > IIRC system was not usable without 'iommu=pt'
>  
> > [  143.347543] sg[0] phys_addr:0x00000003d32e4000 offset:0 length:3072 dma_address:0x00000003d32e4000 dma_length:3072
> > [  143.347547] sg[1] phys_addr:0x00000003d32e4c00 offset:3072 length:65536 dma_address:0x00000003d32e4c00 dma_length:65536
> > [  143.347551] ------------[ cut here ]------------
> > [  143.347552] Invalid SGL for payload:68608 nents:2
> > [  143.347585] WARNING: CPU: 2 PID: 1291 at drivers/nvme/host/pci.c:746
> > [  143.347586] Modules linked in: cfg80211 rfkill aesni_intel crypto_simd cryptd glue_helper fam15h_power k10temp alx mdio i2c_piix4 ohci_pci ohci_hcd snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_hda_core snd_pcm snd_timer sch_fq_codel vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O)
> > [  143.347599] CPU: 2 PID: 1291 Comm: AioMgr1-N Tainted: G           O    T 5.1.1-gentoo #1
> > [  143.347601] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./GA-990FX-GAMING, BIOS NV1 11/03/2015
> > [  143.347603] RIP: 0010:nvme_queue_rq+0xa62/0xad0
> > [  143.347605] Code: 48 c7 c7 d8 86 bf 9a e8 bc 5b d4 ff 41 8b 97 4c 01 00 00 41 f6 47 1e 04 75 59 41 8b 77 24 48 c7 c7 40 3f 38 9a e8 f0 00 92 ff <0f> 0b 41 bc 0a 00 00 00 e9 ed fd ff ff 48 8b 05 5a b3 3f 01 48 85
> > [  143.347606] RSP: 0018:ffffaa9744c8fc10 EFLAGS: 00010282
> > [  143.347607] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000006
> > [  143.347608] RDX: 0000000000000007 RSI: 0000000000000086 RDI: ffff8d3f2ea908d0
> > [  143.347609] RBP: 0000000000000000 R08: ffffaa9744c8fac5 R09: 00000000000003d7
> > [  143.347610] R10: ffffaa9744c8fac0 R11: 0000000000000000 R12: 0000000000000002
> > [  143.347611] R13: ffff8d3f2b69eae8 R14: ffff8d3f2b699158 R15: ffff8d3f2aa7de00
> > [  143.347612] FS:  000071daa321e700(0000) GS:ffff8d3f2ea80000(0000) knlGS:0000000000000000
> > [  143.347613] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [  143.347614] CR2: ffffa10cd510f3c0 CR3: 00000003e3d71000 CR4: 00000000000406e0
> > [  143.347616] Call Trace:
> > [  143.347620]  __blk_mq_try_issue_directly+0x12c/0x1d8
> > [  143.347622]  ? blk_mq_request_issue_directly+0x55/0xf0
> > [  143.347624]  ? blk_mq_try_issue_list_directly+0x4c/0xc0
> > [  143.347626]  ? blk_mq_sched_insert_requests+0x64/0x88
> > [  143.347627]  ? blk_mq_flush_plug_list+0x151/0x190
> > [  143.347629]  ? blk_flush_plug_list+0xea/0x110
> > [  143.347631]  ? blk_finish_plug+0x24/0x32
> > [  143.347633]  ? __x64_sys_io_submit+0xf6/0x168
> > [  143.347635]  ? do_syscall_64+0x46/0xd0
> > [  143.347638]  ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [  143.347639] ---[ end trace 7cb8293d6e867b03 ]---
> 
> [adding Ming, cc linux-block]
> 
> The two elements are physically contiguous, so these should have been
> merged as a single element and we wouldn't have had a problem. The
> following commit looks suspicious:
> 
>   f6970f83ef795 "block: don't check if adjacent bvecs in one bio can be mergeable" 

The two aren't merged because the default segment size(BLK_MAX_SEGMENT_SIZE) is 64KB,
and the following patch may fix this issue:

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index a6644a2c3ef7..c342a23f77f0 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1966,6 +1966,7 @@ static void nvme_set_queue_limits(struct nvme_ctrl *ctrl,
 {
 	bool vwc = false;
 
+	blk_queue_max_segment_size(q, UINT_MAX);
 	if (ctrl->max_hw_sectors) {
 		u32 max_segments =
 			(ctrl->max_hw_sectors / (ctrl->page_size >> 9)) + 1;

Thanks,
Ming