* iSER initiator in 4.5 is unhappy..
@ 2016-04-04 13:46 Christoph Hellwig
[not found] ` <20160404134656.GA30269-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
0 siblings, 1 reply; 20+ messages in thread
From: Christoph Hellwig @ 2016-04-04 13:46 UTC (permalink / raw)
To: sagi-NQWnxTmZq1alnMjI0IkVqw, linux-rdma-u79uwXL29TY76Z2rM5mHXA
When trying to use iSER with a 4.5 initiator it's rather unhappy and
I get a spew of messages like:
[ 236.814107] iser: iser_fast_reg_fmr: ib_fmr_pool_map_phys failed: -22
[ 236.823100] iser: iser_prepare_read_cmd: Failed to set up Data-IN RDMA
[ 236.832101] iser: iser_send_command: conn ffff88002b16e328 failed task->itt 75 err -22
If I manually comment out the FMR selection in iser_memory.c and force
it to use FRs things seem to work fine.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread[parent not found: <20160404134656.GA30269-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>]
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <20160404134656.GA30269-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> @ 2016-04-04 14:42 ` Max Gurtovoy [not found] ` <57027D48.4000500-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2016-04-04 17:29 ` sagig 1 sibling, 1 reply; 20+ messages in thread From: Max Gurtovoy @ 2016-04-04 14:42 UTC (permalink / raw) To: Christoph Hellwig, sagi-NQWnxTmZq1alnMjI0IkVqw, linux-rdma-u79uwXL29TY76Z2rM5mHXA Thanks, I'll try to reprodue it. Can you give me a scenario/setup details ? On 4/4/2016 4:46 PM, Christoph Hellwig wrote: > When trying to use iSER with a 4.5 initiator it's rather unhappy and > I get a spew of messages like: > > [ 236.814107] iser: iser_fast_reg_fmr: ib_fmr_pool_map_phys failed: -22 > [ 236.823100] iser: iser_prepare_read_cmd: Failed to set up Data-IN RDMA > [ 236.832101] iser: iser_send_command: conn ffff88002b16e328 failed task->itt 75 err -22 > > If I manually comment out the FMR selection in iser_memory.c and force > it to use FRs things seem to work fine. > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <57027D48.4000500-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <57027D48.4000500-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> @ 2016-04-04 15:10 ` Christoph Hellwig 0 siblings, 0 replies; 20+ messages in thread From: Christoph Hellwig @ 2016-04-04 15:10 UTC (permalink / raw) To: Max Gurtovoy Cc: Christoph Hellwig, sagi-NQWnxTmZq1alnMjI0IkVqw, linux-rdma-u79uwXL29TY76Z2rM5mHXA LIO target running either 4.5 or 4.6-rc2 backed by an NVMe device on the other side. The systems are connected using Connect-X 3 and 4 devices, and based on the IP address used it should be routed to the Connect-X 3 ones on either side. I did a mkfs.xfs on the device (works) and then started a run of the "quick" xfstests group, which didn't even start output for the first test - based on that it happened very early during mount. FYI, FRs didn't look much better either a few tests into the xfstests run: [ 343.062657] iser: iser_prepare_read_cmd: Failed to set up Data-IN RDMA [ 343.070861] iser: iser_send_command: conn ffff8804a4682328 failed task->itt 6 err -22 [ 343.086357] iser: iser_fast_reg_mr: failed to map sg (68/69) -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <20160404134656.GA30269-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2016-04-04 14:42 ` Max Gurtovoy @ 2016-04-04 17:29 ` sagig [not found] ` <5702A46F.5070109-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 1 sibling, 1 reply; 20+ messages in thread From: sagig @ 2016-04-04 17:29 UTC (permalink / raw) To: Christoph Hellwig, linux-rdma-u79uwXL29TY76Z2rM5mHXA > When trying to use iSER with a 4.5 initiator it's rather unhappy and > I get a spew of messages like: > > [ 236.814107] iser: iser_fast_reg_fmr: ib_fmr_pool_map_phys failed: -22 > [ 236.823100] iser: iser_prepare_read_cmd: Failed to set up Data-IN RDMA > [ 236.832101] iser: iser_send_command: conn ffff88002b16e328 failed task->itt 75 err -22 > > If I manually comment out the FMR selection in iser_memory.c and force > it to use FRs things seem to work fine. Thanks for reporting, This must be the bio splitting code again... Christoph, any chance you can run the sg_gaps I/O generator I sent a while back (Don't yet have access to rdma devices)? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <5702A46F.5070109-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>]
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <5702A46F.5070109-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> @ 2016-04-04 17:38 ` sagig [not found] ` <5702A6AC.9020008-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2016-04-04 22:46 ` Christoph Hellwig 1 sibling, 1 reply; 20+ messages in thread From: sagig @ 2016-04-04 17:38 UTC (permalink / raw) To: Christoph Hellwig, linux-rdma-u79uwXL29TY76Z2rM5mHXA Odd, I see that Ming's patches made it to 4.5... Can you verify that this patches exist in your tree: e827091cb1bc block: merge: get the 1st and last bvec via helpers 25e71a99f10e block: get the 1st and last bvec via helpers e0af29171aa8 block: check virt boundary in bio_will_gap() 7bcd79ac50d9 block: bio: introduce helpers to get the 1st and last bvec -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <5702A6AC.9020008-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>]
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <5702A6AC.9020008-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> @ 2016-04-04 22:58 ` Christoph Hellwig [not found] ` <20160404225831.GA6725-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 0 siblings, 1 reply; 20+ messages in thread From: Christoph Hellwig @ 2016-04-04 22:58 UTC (permalink / raw) To: sagig; +Cc: Christoph Hellwig, linux-rdma-u79uwXL29TY76Z2rM5mHXA On Mon, Apr 04, 2016 at 08:38:52PM +0300, sagig wrote: > Odd, I see that Ming's patches made it to 4.5... > > Can you verify that this patches exist in your tree: > e827091cb1bc block: merge: get the 1st and last bvec via helpers > 25e71a99f10e block: get the 1st and last bvec via helpers > e0af29171aa8 block: check virt boundary in bio_will_gap() > 7bcd79ac50d9 block: bio: introduce helpers to get the 1st and last bvec I've done a clean rebuild because I didn't trust the previous one lingering around, and the syptoms are slightly different now, in that I still get the same spew of messages, but it seems like xfstests is very slowly progressing regardless. Also when testing srp I see similar messages with the same setup: [ 130.896116] scsi host7: ib_srp: Failed to map data (-12) [ 130.896225] scsi host7: ib_srp: Failed to map data (-12) [ 130.896232] scsi host7: ib_srp: Failed to map data (-12) [ 130.896235] scsi host7: ib_srp: Failed to map data (-12) [ 130.896835] scsi host7: ib_srp: Failed to map data (-12) [ 130.993339] scsi host7: ib_srp: Failed to map data (-12) [ 130.993345] scsi host7: ib_srp: Failed to map data (-12) [ 130.993349] scsi host7: ib_srp: Failed to map data (-12) [ 130.993841] scsi host7: ib_srp: Failed to map data (-12) [ 130.993844] scsi host7: ib_srp: Failed to map data (-12) [ 131.035723] scsi host7: ib_srp: Failed to map data (-12) [ 131.058006] scsi host7: ib_srp: Failed to map data (-12) [ 131.058059] scsi host7: ib_srp: Failed to map data (-12) -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <20160404225831.GA6725-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>]
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <20160404225831.GA6725-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> @ 2016-04-05 8:55 ` sagig [not found] ` <57037D8E.4030506-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 0 siblings, 1 reply; 20+ messages in thread From: sagig @ 2016-04-05 8:55 UTC (permalink / raw) To: Christoph Hellwig; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA >> Odd, I see that Ming's patches made it to 4.5... >> >> Can you verify that this patches exist in your tree: >> e827091cb1bc block: merge: get the 1st and last bvec via helpers >> 25e71a99f10e block: get the 1st and last bvec via helpers >> e0af29171aa8 block: check virt boundary in bio_will_gap() >> 7bcd79ac50d9 block: bio: introduce helpers to get the 1st and last bvec > > I've done a clean rebuild because I didn't trust the previous one > lingering around, and the syptoms are slightly different now, in that I > still get the same spew of messages, but it seems like xfstests is > very slowly progressing regardless. > > Also when testing srp I see similar messages with the same setup: > > [ 130.896116] scsi host7: ib_srp: Failed to map data (-12) > [ 130.896225] scsi host7: ib_srp: Failed to map data (-12) > [ 130.896232] scsi host7: ib_srp: Failed to map data (-12) > [ 130.896235] scsi host7: ib_srp: Failed to map data (-12) > [ 130.896835] scsi host7: ib_srp: Failed to map data (-12) > [ 130.993339] scsi host7: ib_srp: Failed to map data (-12) > [ 130.993345] scsi host7: ib_srp: Failed to map data (-12) > [ 130.993349] scsi host7: ib_srp: Failed to map data (-12) > [ 130.993841] scsi host7: ib_srp: Failed to map data (-12) > [ 130.993844] scsi host7: ib_srp: Failed to map data (-12) > [ 131.035723] scsi host7: ib_srp: Failed to map data (-12) > [ 131.058006] scsi host7: ib_srp: Failed to map data (-12) > [ 131.058059] scsi host7: ib_srp: Failed to map data (-12) > This implies that maybe something broke in ib_sg_to_pages: Can you run iser with this patch applied (FR or FMR): -- diff --git a/drivers/infiniband/ulp/iser/iser_memory.c b/drivers/infiniband/ulp/iser/iser_memory.c index 44cc85f206f3..67499d8f6a3a 100644 --- a/drivers/infiniband/ulp/iser/iser_memory.c +++ b/drivers/infiniband/ulp/iser/iser_memory.c @@ -143,7 +143,7 @@ static void iser_data_buf_dump(struct iser_data_buf *data, int i; for_each_sg(data->sg, sg, data->dma_nents, i) - iser_dbg("sg[%d] dma_addr:0x%lX page:0x%p " + iser_err("sg[%d] dma_addr:0x%lX page:0x%p " "off:0x%x sz:0x%x dma_len:0x%x\n", i, (unsigned long)ib_sg_dma_address(ibdev, sg), sg_page(sg), sg->offset, @@ -450,6 +450,7 @@ static int iser_fast_reg_mr(struct iscsi_iser_task *iser_task, if (unlikely(n != mem->size)) { iser_err("failed to map sg (%d/%d)\n", n, mem->size); + iser_data_buf_dump(mem, iser_task->iser_conn->ib_conn.device->ib_device); return n < 0 ? n : -EINVAL; } -- This sg dump will tell us what sort of scatterlist we can't handle. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 20+ messages in thread
[parent not found: <57037D8E.4030506-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>]
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <57037D8E.4030506-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> @ 2016-04-05 9:43 ` Christoph Hellwig [not found] ` <20160405094339.GA1622-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 0 siblings, 1 reply; 20+ messages in thread From: Christoph Hellwig @ 2016-04-05 9:43 UTC (permalink / raw) To: sagig; +Cc: Christoph Hellwig, linux-rdma-u79uwXL29TY76Z2rM5mHXA On Tue, Apr 05, 2016 at 11:55:42AM +0300, sagig wrote: > This implies that maybe something broke in ib_sg_to_pages: > > Can you run iser with this patch applied (FR or FMR): For FMR this won't trigger, but here is the FR run: [ 63.016616] iser: iser_fast_reg_mr: failed to map sg (101/102) [ 63.023183] iser: iser_data_buf_dump: sg[0] dma_addr:0x90E15D200 page:0xffffea0024385740 off:0x200 sz:0xe00 dma_len:0xe00 [ 63.035466] iser: iser_data_buf_dump: sg[1] dma_addr:0x90611D000 page:0xffffea0024184740 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.047749] iser: iser_data_buf_dump: sg[2] dma_addr:0x90EDB3000 page:0xffffea00243b6cc0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.060050] iser: iser_data_buf_dump: sg[3] dma_addr:0x9062CB000 page:0xffffea002418b2c0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.072328] iser: iser_data_buf_dump: sg[4] dma_addr:0x9061EA000 page:0xffffea0024187a80 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.084614] iser: iser_data_buf_dump: sg[5] dma_addr:0x90E7D2000 page:0xffffea002439f480 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.096907] iser: iser_data_buf_dump: sg[6] dma_addr:0x90E5EF000 page:0xffffea0024397bc0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.109195] iser: iser_data_buf_dump: sg[7] dma_addr:0x90B755000 page:0xffffea00242dd540 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.121487] iser: iser_data_buf_dump: sg[8] dma_addr:0x9016F2000 page:0xffffea002405bc80 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.133780] iser: iser_data_buf_dump: sg[9] dma_addr:0x906194000 page:0xffffea0024186500 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.146074] iser: iser_data_buf_dump: sg[10] dma_addr:0x9017A2000 page:0xffffea002405e880 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.158456] iser: iser_data_buf_dump: sg[11] dma_addr:0x92D4BC000 page:0xffffea0024b52f00 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.170841] iser: iser_data_buf_dump: sg[12] dma_addr:0x92D4BF000 page:0xffffea0024b52fc0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.183233] iser: iser_data_buf_dump: sg[13] dma_addr:0x93FA69000 page:0xffffea0024fe9a40 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.195608] iser: iser_data_buf_dump: sg[14] dma_addr:0x90E011000 page:0xffffea0024380440 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.208082] iser: iser_data_buf_dump: sg[15] dma_addr:0x90EACD000 page:0xffffea00243ab340 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.220499] iser: iser_data_buf_dump: sg[16] dma_addr:0x93FA6A000 page:0xffffea0024fe9a80 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.232916] iser: iser_data_buf_dump: sg[17] dma_addr:0x93FD23000 page:0xffffea0024ff48c0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.245334] iser: iser_data_buf_dump: sg[18] dma_addr:0x903D57000 page:0xffffea00240f55c0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.257748] iser: iser_data_buf_dump: sg[19] dma_addr:0x93FF38000 page:0xffffea0024ffce00 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.270167] iser: iser_data_buf_dump: sg[20] dma_addr:0x90F035000 page:0xffffea00243c0d40 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.282584] iser: iser_data_buf_dump: sg[21] dma_addr:0x901719000 page:0xffffea002405c640 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.295000] iser: iser_data_buf_dump: sg[22] dma_addr:0x90EBA3000 page:0xffffea00243ae8c0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.307422] iser: iser_data_buf_dump: sg[23] dma_addr:0x90E20B000 page:0xffffea00243882c0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.319837] iser: iser_data_buf_dump: sg[24] dma_addr:0x901722000 page:0xffffea002405c880 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.332255] iser: iser_data_buf_dump: sg[25] dma_addr:0x93FA52000 page:0xffffea0024fe9480 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.344679] iser: iser_data_buf_dump: sg[26] dma_addr:0x90E023000 page:0xffffea00243808c0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.357095] iser: iser_data_buf_dump: sg[27] dma_addr:0x901679000 page:0xffffea0024059e40 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.369518] iser: iser_data_buf_dump: sg[28] dma_addr:0x90DC4A000 page:0xffffea0024371280 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.381937] iser: iser_data_buf_dump: sg[29] dma_addr:0x90160D000 page:0xffffea0024058340 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.394353] iser: iser_data_buf_dump: sg[30] dma_addr:0x90E9F3000 page:0xffffea00243a7cc0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.406769] iser: iser_data_buf_dump: sg[31] dma_addr:0x90E6CA000 page:0xffffea002439b280 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.419175] iser: iser_data_buf_dump: sg[32] dma_addr:0x90E6DA000 page:0xffffea002439b680 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.431596] iser: iser_data_buf_dump: sg[33] dma_addr:0x901787000 page:0xffffea002405e1c0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.444013] iser: iser_data_buf_dump: sg[34] dma_addr:0x90E6A7000 page:0xffffea002439a9c0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.456430] iser: iser_data_buf_dump: sg[35] dma_addr:0x90178C000 page:0xffffea002405e300 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.468853] iser: iser_data_buf_dump: sg[36] dma_addr:0x90E27A000 page:0xffffea0024389e80 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.481278] iser: iser_data_buf_dump: sg[37] dma_addr:0x90E6CC000 page:0xffffea002439b300 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.493702] iser: iser_data_buf_dump: sg[38] dma_addr:0x90E6CF000 page:0xffffea002439b3c0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.506116] iser: iser_data_buf_dump: sg[39] dma_addr:0x901790000 page:0xffffea002405e400 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.518531] iser: iser_data_buf_dump: sg[40] dma_addr:0x90E6D0000 page:0xffffea002439b400 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.530955] iser: iser_data_buf_dump: sg[41] dma_addr:0x90E6D6000 page:0xffffea002439b580 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.543370] iser: iser_data_buf_dump: sg[42] dma_addr:0x92D266000 page:0xffffea0024b49980 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.555778] iser: iser_data_buf_dump: sg[43] dma_addr:0x90177B000 page:0xffffea002405dec0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.568886] iser: iser_data_buf_dump: sg[44] dma_addr:0x92D265000 page:0xffffea0024b49940 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.581950] iser: iser_data_buf_dump: sg[45] dma_addr:0x90E240000 page:0xffffea0024389000 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.595005] iser: iser_data_buf_dump: sg[46] dma_addr:0x901635000 page:0xffffea0024058d40 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.608022] iser: iser_data_buf_dump: sg[47] dma_addr:0x9062B8000 page:0xffffea002418ae00 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.620986] iser: iser_data_buf_dump: sg[48] dma_addr:0x90EACF000 page:0xffffea00243ab3c0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.633939] iser: iser_data_buf_dump: sg[49] dma_addr:0x901631000 page:0xffffea0024058c40 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.646887] iser: iser_data_buf_dump: sg[50] dma_addr:0x90E8DE000 page:0xffffea00243a3780 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.659842] iser: iser_data_buf_dump: sg[51] dma_addr:0x9034F4000 page:0xffffea00240d3d00 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.672794] iser: iser_data_buf_dump: sg[52] dma_addr:0x903CF7000 page:0xffffea00240f3dc0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.685744] iser: iser_data_buf_dump: sg[53] dma_addr:0x90E511000 page:0xffffea0024394440 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.698371] iser: iser_data_buf_dump: sg[54] dma_addr:0x90F328000 page:0xffffea00243cca00 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.711008] iser: iser_data_buf_dump: sg[55] dma_addr:0x90E3BC000 page:0xffffea002438ef00 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.723643] iser: iser_data_buf_dump: sg[56] dma_addr:0x90F32E000 page:0xffffea00243ccb80 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.736277] iser: iser_data_buf_dump: sg[57] dma_addr:0x90F323000 page:0xffffea00243cc8c0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.748893] iser: iser_data_buf_dump: sg[58] dma_addr:0x9064CD000 page:0xffffea0024193340 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.761509] iser: iser_data_buf_dump: sg[59] dma_addr:0x90169F000 page:0xffffea002405a7c0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.774127] iser: iser_data_buf_dump: sg[60] dma_addr:0x9062B4000 page:0xffffea002418ad00 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.786744] iser: iser_data_buf_dump: sg[61] dma_addr:0x90310A000 page:0xffffea00240c4280 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.799371] iser: iser_data_buf_dump: sg[62] dma_addr:0x93FE24000 page:0xffffea0024ff8900 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.811990] iser: iser_data_buf_dump: sg[63] dma_addr:0x9063BA000 page:0xffffea002418ee80 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.824608] iser: iser_data_buf_dump: sg[64] dma_addr:0x903CFC000 page:0xffffea00240f3f00 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.837228] iser: iser_data_buf_dump: sg[65] dma_addr:0x903106000 page:0xffffea00240c4180 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.849835] iser: iser_data_buf_dump: sg[66] dma_addr:0x90B93F000 page:0xffffea00242e4fc0 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.862455] iser: iser_data_buf_dump: sg[67] dma_addr:0x90E9C9000 page:0xffffea00243a7240 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.875073] iser: iser_data_buf_dump: sg[68] dma_addr:0x90176D000 page:0xffffea002405db40 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.887689] iser: iser_data_buf_dump: sg[69] dma_addr:0x90177E000 page:0xffffea002405df80 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.900308] iser: iser_data_buf_dump: sg[70] dma_addr:0x90E501000 page:0xffffea0024394040 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.912926] iser: iser_data_buf_dump: sg[71] dma_addr:0x9016FE000 page:0xffffea002405bf80 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.925549] iser: iser_data_buf_dump: sg[72] dma_addr:0x9014F5000 page:0xffffea0024053d40 off:0x0 sz:0x1000 dma_len:0x1000 [ 63.938165] iser: iser_data_buf_dump: sg[73] dma_addr:0x90EB96000 page:0xffffea00243ae580 off:0x0 sz:0x2000 dma_len:0x2000 [ 63.950779] iser: iser_data_buf_dump: sg[74] dma_addr:0x90E22E000 page:0xffffea0024388b80 off:0x0 sz:0x2000 dma_len:0x2000 [ 63.963400] iser: iser_data_buf_dump: sg[75] dma_addr:0x901636000 page:0xffffea0024058d80 off:0x0 sz:0x2000 dma_len:0x2000 [ 63.976015] iser: iser_data_buf_dump: sg[76] dma_addr:0x90169C000 page:0xffffea002405a700 off:0x0 sz:0x2000 dma_len:0x2000 [ 63.988629] iser: iser_data_buf_dump: sg[77] dma_addr:0x90163A000 page:0xffffea0024058e80 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.001238] iser: iser_data_buf_dump: sg[78] dma_addr:0x90177C000 page:0xffffea002405df00 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.013852] iser: iser_data_buf_dump: sg[79] dma_addr:0x903FBE000 page:0xffffea00240fef80 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.026462] iser: iser_data_buf_dump: sg[80] dma_addr:0x9061B6000 page:0xffffea0024186d80 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.039077] iser: iser_data_buf_dump: sg[81] dma_addr:0x90E7FA000 page:0xffffea002439fe80 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.051697] iser: iser_data_buf_dump: sg[82] dma_addr:0x90E9D4000 page:0xffffea00243a7500 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.064310] iser: iser_data_buf_dump: sg[83] dma_addr:0x90E502000 page:0xffffea0024394080 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.076933] iser: iser_data_buf_dump: sg[84] dma_addr:0x901686000 page:0xffffea002405a180 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.089548] iser: iser_data_buf_dump: sg[85] dma_addr:0x903458000 page:0xffffea00240d1600 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.102163] iser: iser_data_buf_dump: sg[86] dma_addr:0x90E43A000 page:0xffffea0024390e80 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.114779] iser: iser_data_buf_dump: sg[87] dma_addr:0x90F190000 page:0xffffea00243c6400 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.127395] iser: iser_data_buf_dump: sg[88] dma_addr:0x9016F6000 page:0xffffea002405bd80 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.140002] iser: iser_data_buf_dump: sg[89] dma_addr:0x92D3AC000 page:0xffffea0024b4eb00 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.152621] iser: iser_data_buf_dump: sg[90] dma_addr:0x903F86000 page:0xffffea00240fe180 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.165239] iser: iser_data_buf_dump: sg[91] dma_addr:0x901598000 page:0xffffea0024056600 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.177857] iser: iser_data_buf_dump: sg[92] dma_addr:0x90F7D4000 page:0xffffea00243df500 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.190476] iser: iser_data_buf_dump: sg[93] dma_addr:0x90E31E000 page:0xffffea002438c780 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.203147] iser: iser_data_buf_dump: sg[94] dma_addr:0x90E31A000 page:0xffffea002438c680 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.215764] iser: iser_data_buf_dump: sg[95] dma_addr:0x90E440000 page:0xffffea0024391000 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.228380] iser: iser_data_buf_dump: sg[96] dma_addr:0x90E43E000 page:0xffffea0024390f80 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.240995] iser: iser_data_buf_dump: sg[97] dma_addr:0x901586000 page:0xffffea0024056180 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.253611] iser: iser_data_buf_dump: sg[98] dma_addr:0x90F390000 page:0xffffea00243ce400 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.266219] iser: iser_data_buf_dump: sg[99] dma_addr:0x90660E000 page:0xffffea0024198380 off:0x0 sz:0x2000 dma_len:0x2000 [ 64.278836] iser: iser_data_buf_dump: sg[100] dma_addr:0x9017E6000 page:0xffffea002405f980 off:0x0 sz:0x1000 dma_len:0x1000 [ 64.291577] iser: iser_data_buf_dump: sg[101] dma_addr:0x93FBF1000 page:0xffffea0024fefc40 off:0x0 sz:0x200 dma_len:0x200 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <20160405094339.GA1622-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>]
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <20160405094339.GA1622-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> @ 2016-04-06 17:34 ` sagig [not found] ` <570548AB.4090802-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 0 siblings, 1 reply; 20+ messages in thread From: sagig @ 2016-04-06 17:34 UTC (permalink / raw) To: Christoph Hellwig; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA >> This implies that maybe something broke in ib_sg_to_pages: >> >> Can you run iser with this patch applied (FR or FMR): > > For FMR this won't trigger, but here is the FR run: > > [ 63.016616] iser: iser_fast_reg_mr: failed to map sg (101/102) > [ 63.023183] iser: iser_data_buf_dump: sg[0] dma_addr:0x90E15D200 > page:0xffffea0024385740 off:0x200 sz:0xe00 dma_len:0xe00 > [ 63.035466] iser: iser_data_buf_dump: sg[1] dma_addr:0x90611D000 > page:0xffffea0024184740 off:0x0 sz:0x1000 dma_len:0x1000 > [ 63.047749] iser: iser_data_buf_dump: sg[2] dma_addr:0x90EDB3000 > page:0xffffea00243b6cc0 off:0x0 sz:0x1000 dma_len:0x1000 > [ 63.060050] iser: iser_data_buf_dump: sg[3] dma_addr:0x9062CB000 > page:0xffffea002418b2c0 off:0x0 sz:0x1000 dma_len:0x1000 ... > [ 64.215764] iser: iser_data_buf_dump: sg[95] dma_addr:0x90E440000 > page:0xffffea0024391000 off:0x0 sz:0x2000 dma_len:0x2000 > [ 64.228380] iser: iser_data_buf_dump: sg[96] dma_addr:0x90E43E000 > page:0xffffea0024390f80 off:0x0 sz:0x2000 dma_len:0x2000 > [ 64.240995] iser: iser_data_buf_dump: sg[97] dma_addr:0x901586000 > page:0xffffea0024056180 off:0x0 sz:0x2000 dma_len:0x2000 > [ 64.253611] iser: iser_data_buf_dump: sg[98] dma_addr:0x90F390000 > page:0xffffea00243ce400 off:0x0 sz:0x2000 dma_len:0x2000 > [ 64.266219] iser: iser_data_buf_dump: sg[99] dma_addr:0x90660E000 > page:0xffffea0024198380 off:0x0 sz:0x2000 dma_len:0x2000 > [ 64.278836] iser: iser_data_buf_dump: sg[100] dma_addr:0x9017E6000 > page:0xffffea002405f980 off:0x0 sz:0x1000 dma_len:0x1000 > [ 64.291577] iser: iser_data_buf_dump: sg[101] dma_addr:0x93FBF1000 > page:0xffffea0024fefc40 off:0x0 sz:0x200 dma_len:0x200 > So this scatterlist seems perfectly fine (no gaps). The only thing I can think of is that we don't have enough space reserved, but I counted 127 pages in the scatterlist when iser reserves 128 (by default). Just in case I mis-counted, can you (or Max) try with the below patch: -- diff --git a/drivers/infiniband/ulp/iser/iser_initiator.c b/drivers/infiniband/ulp/iser/iser_initiator.c index ed54b388e7ad..f2ac891c432a 100644 --- a/drivers/infiniband/ulp/iser/iser_initiator.c +++ b/drivers/infiniband/ulp/iser/iser_initiator.c @@ -250,7 +250,7 @@ int iser_alloc_rx_descriptors(struct iser_conn *iser_conn, iser_conn->min_posted_rx = iser_conn->qp_max_recv_dtos >> 2; if (device->reg_ops->alloc_reg_res(ib_conn, session->scsi_cmds_max, - iser_conn->scsi_sg_tablesize)) + iser_conn->scsi_sg_tablesize + 1)) goto create_rdma_reg_res_failed; if (iser_alloc_login_buf(iser_conn)) diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c b/drivers/infiniband/ulp/iser/iser_verbs.c index 40c0f4978e2f..937b97f34be1 100644 --- a/drivers/infiniband/ulp/iser/iser_verbs.c +++ b/drivers/infiniband/ulp/iser/iser_verbs.c @@ -721,7 +721,7 @@ iser_calc_scsi_params(struct iser_conn *iser_conn, sg_tablesize = DIV_ROUND_UP(max_sectors * 512, SIZE_4K); sup_sg_tablesize = min_t(unsigned, ISCSI_ISER_MAX_SG_TABLESIZE, - device->ib_device->attrs.max_fast_reg_page_list_len); + device->ib_device->attrs.max_fast_reg_page_list_len - 1); if (sg_tablesize > sup_sg_tablesize) { sg_tablesize = sup_sg_tablesize; -- -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 20+ messages in thread
[parent not found: <570548AB.4090802-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>]
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <570548AB.4090802-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> @ 2016-04-06 23:16 ` Christoph Hellwig [not found] ` <20160406231658.GB11588-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 0 siblings, 1 reply; 20+ messages in thread From: Christoph Hellwig @ 2016-04-06 23:16 UTC (permalink / raw) To: sagig; +Cc: Christoph Hellwig, linux-rdma-u79uwXL29TY76Z2rM5mHXA On Wed, Apr 06, 2016 at 08:34:35PM +0300, sagig wrote: > So this scatterlist seems perfectly fine (no gaps). The only thing I > can think of is that we don't have enough space reserved, but I counted > 127 pages in the scatterlist when iser reserves 128 (by default). > > Just in case I mis-counted, can you (or Max) try with the below > patch: This works fine for me, but I wonder if the miscounting is in the block layer? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <20160406231658.GB11588-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>]
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <20160406231658.GB11588-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> @ 2016-04-07 7:14 ` Sagi Grimberg [not found] ` <570608C9.1070805-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 0 siblings, 1 reply; 20+ messages in thread From: Sagi Grimberg @ 2016-04-07 7:14 UTC (permalink / raw) To: Christoph Hellwig; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA >> So this scatterlist seems perfectly fine (no gaps). The only thing I >> can think of is that we don't have enough space reserved, but I counted >> 127 pages in the scatterlist when iser reserves 128 (by default). >> >> Just in case I mis-counted, can you (or Max) try with the below >> patch: > > This works fine for me, but I wonder if the miscounting is in the > block layer? We communicate the "max_pages" via sg_tablesize and max_hw_sectors. iSER communicates (or at least supposed to) sg_tablesize=128 and max_sectors=1024 to scsi (which sets the block queues attributes), that should be sufficient for handling all the incoming IO. Unless we didn't account for the first-page offset and the reminder coming at the last segment (which results in an extra page). The patch should fix the problem, however in this particular sg list I don't see the violation (counted 127 pages)... -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <570608C9.1070805-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>]
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <570608C9.1070805-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> @ 2016-04-07 9:09 ` Max Gurtovoy [not found] ` <570623B9.8040108-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 0 siblings, 1 reply; 20+ messages in thread From: Max Gurtovoy @ 2016-04-07 9:09 UTC (permalink / raw) To: Sagi Grimberg, Christoph Hellwig; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA On 4/7/2016 10:14 AM, Sagi Grimberg wrote: > >>> So this scatterlist seems perfectly fine (no gaps). The only thing I >>> can think of is that we don't have enough space reserved, but I counted >>> 127 pages in the scatterlist when iser reserves 128 (by default). >>> >>> Just in case I mis-counted, can you (or Max) try with the below >>> patch: >> >> This works fine for me, but I wonder if the miscounting is in the >> block layer? > > We communicate the "max_pages" via sg_tablesize and max_hw_sectors. > > iSER communicates (or at least supposed to) sg_tablesize=128 and > max_sectors=1024 to scsi (which sets the block queues attributes), that > should be sufficient for handling all the incoming IO. Unless we didn't > account for the first-page offset and the reminder coming at the last > segment (which results in an extra page). The patch should fix the > problem, however in this particular sg list I don't see the violation > (counted 127 pages)... I counted 129 pages. I will run some tests in our labs too. Thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <570623B9.8040108-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <570623B9.8040108-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> @ 2016-04-07 15:16 ` Sagi Grimberg [not found] ` <570679B6.10409-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 0 siblings, 1 reply; 20+ messages in thread From: Sagi Grimberg @ 2016-04-07 15:16 UTC (permalink / raw) To: Max Gurtovoy, Christoph Hellwig; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA >> >> We communicate the "max_pages" via sg_tablesize and max_hw_sectors. >> >> iSER communicates (or at least supposed to) sg_tablesize=128 and >> max_sectors=1024 to scsi (which sets the block queues attributes), that >> should be sufficient for handling all the incoming IO. Unless we didn't >> account for the first-page offset and the reminder coming at the last >> segment (which results in an extra page). The patch should fix the >> problem, however in this particular sg list I don't see the violation >> (counted 127 pages)... > > I counted 129 pages. > I will run some tests in our labs too. If you did then that indicates the block layer violates its obligation to respect max_segments/max_hw_sectors? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <570679B6.10409-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>]
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <570679B6.10409-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> @ 2016-04-07 15:20 ` Christoph Hellwig 2016-04-07 22:13 ` Christoph Hellwig 1 sibling, 0 replies; 20+ messages in thread From: Christoph Hellwig @ 2016-04-07 15:20 UTC (permalink / raw) To: Sagi Grimberg Cc: Max Gurtovoy, Christoph Hellwig, linux-rdma-u79uwXL29TY76Z2rM5mHXA On Thu, Apr 07, 2016 at 06:16:06PM +0300, Sagi Grimberg wrote: > >>segment (which results in an extra page). The patch should fix the > >>problem, however in this particular sg list I don't see the violation > >>(counted 127 pages)... > > > >I counted 129 pages. > >I will run some tests in our labs too. > > If you did then that indicates the block layer violates > its obligation to respect max_segments/max_hw_sectors? I spent some time looking over the code and remembered an issue I fixed for NVMf (which might explain why we're not seeing it there). max_segments just means contiguous segments, not nessecarily 4k pages. For that we either need a small enough max_sectors_hw or set a max_segment_size of 4k, otherwise we might get 128 block segments, but due to lacking alignment they'll expand to 129 pages. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <570679B6.10409-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2016-04-07 15:20 ` Christoph Hellwig @ 2016-04-07 22:13 ` Christoph Hellwig [not found] ` <20160407221359.GA16686-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 1 sibling, 1 reply; 20+ messages in thread From: Christoph Hellwig @ 2016-04-07 22:13 UTC (permalink / raw) To: Sagi Grimberg Cc: Max Gurtovoy, bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ, linux-rdma-u79uwXL29TY76Z2rM5mHXA The patch below fixes the issue for me. But for SRP I'm still seeing lots of these failures even with the equivalent patch applied: [ 426.027905] scsi host8: ib_srp: Failed to map data (-12) [ 426.053203] scsi host8: ib_srp: Failed to map data (-12) [ 426.053265] scsi host8: ib_srp: Failed to map data (-12) [ 426.053270] scsi host8: ib_srp: Failed to map data (-12) [ 426.053273] scsi host8: ib_srp: Failed to map data (-12) [ 426.053276] scsi host8: ib_srp: Failed to map data (-12) [ 426.053279] scsi host8: ib_srp: Failed to map data (-12) It seems the actual reproducer is xfs_repair as run by xfstests. diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index 60b169a..3ac4755 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.c +++ b/drivers/infiniband/ulp/srp/ib_srp.c @@ -3205,6 +3205,7 @@ static ssize_t srp_create_target(struct device *dev, target_host->max_id = 1; target_host->max_lun = -1LL; target_host->max_cmd_len = sizeof ((struct srp_cmd *) (void *) 0L)->cdb; + target_host->max_segment_size = PAGE_SIZE; target = host_to_target(target_host); diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index 8c6e318..ae331e0 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -2120,7 +2120,8 @@ static void __scsi_init_queue(struct Scsi_Host *shost, struct request_queue *q) blk_queue_segment_boundary(q, shost->dma_boundary); dma_set_seg_boundary(dev, shost->dma_boundary); - blk_queue_max_segment_size(q, dma_get_max_seg_size(dev)); + blk_queue_max_segment_size(q, + min(shost->max_segment_size, dma_get_max_seg_size(dev))); if (!shost->use_clustering) q->limits.cluster = 0; diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h index fcfa3d7..f11d3fe 100644 --- a/include/scsi/scsi_host.h +++ b/include/scsi/scsi_host.h @@ -621,6 +621,7 @@ struct Scsi_Host { short unsigned int sg_tablesize; short unsigned int sg_prot_tablesize; unsigned int max_sectors; + unsigned int max_segment_size; unsigned long dma_boundary; /* * In scsi-mq mode, the number of hardware queues supported by the LLD. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 20+ messages in thread
[parent not found: <20160407221359.GA16686-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>]
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <20160407221359.GA16686-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> @ 2016-04-08 1:42 ` Bart Van Assche 2016-04-10 13:30 ` Max Gurtovoy 2016-04-12 21:33 ` Bart Van Assche 2 siblings, 0 replies; 20+ messages in thread From: Bart Van Assche @ 2016-04-08 1:42 UTC (permalink / raw) To: Christoph Hellwig, Sagi Grimberg Cc: Max Gurtovoy, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org On 04/07/16 15:14, Christoph Hellwig wrote: > The patch below fixes the issue for me. But for SRP I'm still seeing > lots of these failures even with the equivalent patch applied: > > [ 426.027905] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053203] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053265] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053270] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053273] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053276] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053279] scsi host8: ib_srp: Failed to map data (-12) > > It seems the actual reproducer is xfs_repair as run by xfstests. Hello Christoph, Thanks for the report. I will try to reproduce this behavior and will see whether I can find the root cause. Bart. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <20160407221359.GA16686-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2016-04-08 1:42 ` Bart Van Assche @ 2016-04-10 13:30 ` Max Gurtovoy [not found] ` <570A5563.5060104-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2016-04-12 21:33 ` Bart Van Assche 2 siblings, 1 reply; 20+ messages in thread From: Max Gurtovoy @ 2016-04-10 13:30 UTC (permalink / raw) To: Christoph Hellwig, Sagi Grimberg Cc: bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ, linux-rdma-u79uwXL29TY76Z2rM5mHXA On 4/8/2016 1:13 AM, Christoph Hellwig wrote: > The patch below fixes the issue for me. But for SRP I'm still seeing > lots of these failures even with the equivalent patch applied: > > [ 426.027905] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053203] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053265] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053270] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053273] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053276] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053279] scsi host8: ib_srp: Failed to map data (-12) > > It seems the actual reproducer is xfs_repair as run by xfstests. > > diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c > index 60b169a..3ac4755 100644 > --- a/drivers/infiniband/ulp/srp/ib_srp.c > +++ b/drivers/infiniband/ulp/srp/ib_srp.c > @@ -3205,6 +3205,7 @@ static ssize_t srp_create_target(struct device *dev, > target_host->max_id = 1; > target_host->max_lun = -1LL; > target_host->max_cmd_len = sizeof ((struct srp_cmd *) (void *) 0L)->cdb; > + target_host->max_segment_size = PAGE_SIZE; > > target = host_to_target(target_host); Hi Christoph, did you add similar assignment in iser code or only used the scsi patches ? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <570A5563.5060104-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <570A5563.5060104-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> @ 2016-04-10 14:52 ` Christoph Hellwig 0 siblings, 0 replies; 20+ messages in thread From: Christoph Hellwig @ 2016-04-10 14:52 UTC (permalink / raw) To: Max Gurtovoy Cc: Christoph Hellwig, Sagi Grimberg, bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ, linux-rdma-u79uwXL29TY76Z2rM5mHXA On Sun, Apr 10, 2016 at 04:30:11PM +0300, Max Gurtovoy wrote: > Hi Christoph, > did you add similar assignment in iser code or only used the scsi patches ? Yes, to fix it I had to do the assignment in iser. Not sure why I didn't manage to send it out. Let me prepare a proper patch series. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <20160407221359.GA16686-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2016-04-08 1:42 ` Bart Van Assche 2016-04-10 13:30 ` Max Gurtovoy @ 2016-04-12 21:33 ` Bart Van Assche 2 siblings, 0 replies; 20+ messages in thread From: Bart Van Assche @ 2016-04-12 21:33 UTC (permalink / raw) To: Christoph Hellwig, Sagi Grimberg Cc: Max Gurtovoy, bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ, linux-rdma-u79uwXL29TY76Z2rM5mHXA On 04/07/2016 03:13 PM, Christoph Hellwig wrote: > The patch below fixes the issue for me. But for SRP I'm still seeing > lots of these failures even with the equivalent patch applied: > > [ 426.027905] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053203] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053265] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053270] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053273] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053276] scsi host8: ib_srp: Failed to map data (-12) > [ 426.053279] scsi host8: ib_srp: Failed to map data (-12) > > It seems the actual reproducer is xfs_repair as run by xfstests. Hello Christoph, So far I have not yet been able to reproduce the ib_srp mapping failure with kernel v4.6-rc3. How was ib_srp configured in your test (cat /etc/modprobe.d/ib_srp.conf)? Thanks, Bart. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: iSER initiator in 4.5 is unhappy.. [not found] ` <5702A46F.5070109-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2016-04-04 17:38 ` sagig @ 2016-04-04 22:46 ` Christoph Hellwig 1 sibling, 0 replies; 20+ messages in thread From: Christoph Hellwig @ 2016-04-04 22:46 UTC (permalink / raw) To: sagig; +Cc: Christoph Hellwig, linux-rdma-u79uwXL29TY76Z2rM5mHXA On Mon, Apr 04, 2016 at 08:29:19PM +0300, sagig wrote: > Christoph, any chance you can run the sg_gaps I/O generator > I sent a while back (Don't yet have access to rdma devices)? A run as ./sg-gaps -d /dev/sdb -s 16 -n 8 -v doesn't show any errors, anything else I should try? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2016-04-12 21:33 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-04-04 13:46 iSER initiator in 4.5 is unhappy Christoph Hellwig
[not found] ` <20160404134656.GA30269-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2016-04-04 14:42 ` Max Gurtovoy
[not found] ` <57027D48.4000500-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2016-04-04 15:10 ` Christoph Hellwig
2016-04-04 17:29 ` sagig
[not found] ` <5702A46F.5070109-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2016-04-04 17:38 ` sagig
[not found] ` <5702A6AC.9020008-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2016-04-04 22:58 ` Christoph Hellwig
[not found] ` <20160404225831.GA6725-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2016-04-05 8:55 ` sagig
[not found] ` <57037D8E.4030506-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2016-04-05 9:43 ` Christoph Hellwig
[not found] ` <20160405094339.GA1622-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2016-04-06 17:34 ` sagig
[not found] ` <570548AB.4090802-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2016-04-06 23:16 ` Christoph Hellwig
[not found] ` <20160406231658.GB11588-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2016-04-07 7:14 ` Sagi Grimberg
[not found] ` <570608C9.1070805-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2016-04-07 9:09 ` Max Gurtovoy
[not found] ` <570623B9.8040108-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2016-04-07 15:16 ` Sagi Grimberg
[not found] ` <570679B6.10409-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2016-04-07 15:20 ` Christoph Hellwig
2016-04-07 22:13 ` Christoph Hellwig
[not found] ` <20160407221359.GA16686-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2016-04-08 1:42 ` Bart Van Assche
2016-04-10 13:30 ` Max Gurtovoy
[not found] ` <570A5563.5060104-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2016-04-10 14:52 ` Christoph Hellwig
2016-04-12 21:33 ` Bart Van Assche
2016-04-04 22:46 ` Christoph Hellwig
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.