Re: [PATCH v7] NVMe: conversion to blk-mq

Keith Busch
2014-06-10T19:29:36Z

[-- Attachment #1: Type: TEXT/PLAIN, Size: 5073 bytes --]

On Tue, 10 Jun 2014, Jens Axboe wrote:
>> On Jun 10, 2014, at 9:52 AM, Keith Busch <keith.busch@intel.com> wrote:
>>
>>> On Tue, 10 Jun 2014, Matias Bjørling wrote:
>>> This converts the current NVMe driver to utilize the blk-mq layer.
>>
>> I'd like to run xfstests on this, but it is failing mkfs.xfs. I honestly
>> don't know much about this area, but I think this may be from the recent
>> chunk sectors patch causing a __bio_add_page to reject adding a new page.
>
> Gah, yes that's a bug in the chunk patch. It must always allow a single page
> at any offset. I'll test and send out a fix.

I have two devices, one formatted 4k, the other 512. The 4k is used as
the TEST_DEV and 512 is used as SCRATCH_DEV. I'm always hitting a BUG when
unmounting the scratch dev in xfstests generic/068. The bug looks like
nvme was trying to use an SGL that doesn't map correctly to a PRP.

Also, it doesn't look like this driver can recover from an unresponsive
device, leaving tasks in uniterruptible sleep state forever. Still looking
into that one though; as far as I can tell the device is perfectly fine,
but lots of "Cancelling I/O" messages are getting logged.

[ 478.580863] ------------[ cut here ]------------
[ 478.586111] kernel BUG at drivers/block/nvme-core.c:486!
[ 478.592130] invalid opcode: 0000 [#1] SMP
[ 478.596963] Modules linked in: xfs nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp llc jfs joydev hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd microcode ehci_pci ehci_hcd pcspkr usbcore acpi_cpufreq lpc_ich ioatdma mfd_core usb_common i2c_i801 evdev wmi tpm_tis ipmi_si tpm ipmi_msghandler processor thermal_sys button ext4 crc16 jbd2 mbcache dm_mod nbd sg sd_mod sr_mod crc_t10dif cdrom crct10dif_common crc32c_intel isci libsas ahci libahci scsi_transport_sas igb libata ptp pps_core i2c_algo_bit scsi_mod i2c_core dca [last unloaded: nvme]
[ 478.682913] CPU: 5 PID: 17969 Comm: fsstress Not tainted 3.15.0-rc8+ #19
[ 478.690510] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
[ 478.702126] task: ffff88042bc18cf0 ti: ffff88042d3f0000 task.ti: ffff88042d3f0000
[ 478.710624] RIP: 0010:[<ffffffffa04bc8e6>] [<ffffffffa04bc8e6>] nvme_setup_prps+0x1b8/0x1eb [nvme]
[ 478.720971] RSP: 0018:ffff88042d3f38c8 EFLAGS: 00010286
[ 478.727013] RAX: 0000000000000014 RBX: ffff88042b96e400 RCX: 0000000803d0e000
[ 478.735096] RDX: 0000000000000015 RSI: 0000000000000246 RDI: ffff88042b96e6b0
[ 478.743177] RBP: 0000000000015e00 R08: 0000000000000000 R09: 0000000000000e00
[ 478.751264] R10: 0000000000000e00 R11: ffff88042d3f3900 R12: ffff88042b96e6d0
[ 478.759349] R13: ffff880823f40e00 R14: ffff88042b96e710 R15: 00000000fffffc00
[ 478.767435] FS: 00007f92eb29c700(0000) GS:ffff88043f6a0000(0000) knlGS:0000000000000000
[ 478.776614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 478.783143] CR2: 00007f92e401ff18 CR3: 000000042b5d5000 CR4: 00000000000407e0
[ 478.791218] Stack:
[ 478.793558] ffff8808229367c0 0000000805205000 ffff880400000014 00000a0029abf540
[ 478.802302] ffff88082bcd8140 0000031000000020 0000000000000017 0000000823f40e00
[ 478.811045] 0000000000000e00 ffff880827de3300 ffff88042b96e400 ffff88082ad60c40
[ 478.819789] Call Trace:
[ 478.822630] [<ffffffffa04bca5e>] ? nvme_queue_rq+0x145/0x33b [nvme]
[ 478.829859] [<ffffffff811c854f>] ? blk_mq_make_request+0xd7/0x140
[ 478.836891] [<ffffffff811bf583>] ? generic_make_request+0x98/0xd5
[ 478.843906] [<ffffffff811c0240>] ? submit_bio+0x100/0x109
[ 478.850161] [<ffffffff81142bc2>] ? dio_bio_submit+0x67/0x86
[ 478.856596] [<ffffffff81143a08>] ? do_blockdev_direct_IO+0x956/0xad8
[ 478.863924] [<ffffffffa0592a2e>] ? __xfs_get_blocks+0x410/0x410 [xfs]
[ 478.871338] [<ffffffffa0591c12>] ? xfs_vm_direct_IO+0xda/0x146 [xfs]
[ 478.878652] [<ffffffffa0592a2e>] ? __xfs_get_blocks+0x410/0x410 [xfs]
[ 478.886066] [<ffffffffa0592b00>] ? xfs_finish_ioend_sync+0x1a/0x1a [xfs]
[ 478.893775] [<ffffffff810d1749>] ? generic_file_direct_write+0xe2/0x145
[ 478.901385] [<ffffffffa05e81c0>] ? xfs_file_dio_aio_write+0x1ba/0x208 [xfs]
[ 478.909391] [<ffffffffa059c43d>] ? xfs_file_aio_write+0xc4/0x157 [xfs]
[ 478.916892] [<ffffffff8111801a>] ? do_sync_write+0x50/0x73
[ 478.923227] [<ffffffff81118b36>] ? vfs_write+0x9f/0xfc
[ 478.929173] [<ffffffff81118e22>] ? SyS_write+0x56/0x8a
[ 478.935122] [<ffffffff8139fe52>] ? system_call_fastpath+0x16/0x1b
[ 478.942137] Code: 48 63 c2 41 81 ef 00 10 00 00 ff c2 83 7c 24 1c 00 49 89 4c c5 00 7e 35 48 81 c1 00 10 00 00 41 83 ff 00 0f 8f 6f ff ff ff 74 02 <0f> 0b 4c 89 e7 89 54 24 10 e8 03 e3 d2 e0 8b 54 24 10 49 89 c4
[ 478.968952] RIP [<ffffffffa04bc8e6>] nvme_setup_prps+0x1b8/0x1eb [nvme]
[ 478.976638] RSP <ffff88042d3f38c8>
[ 478.980699] ---[ end trace 3323c3dc4ef42ff8 ]---