iSER Connection via LIO not working

iSER Connection via LIO not working

[Thomas Rosenstein]

Hello,

i have setup a lio with 1 LUN on kernel 4.9.11, works fine via iSCSI, 
but when switching to iSER the message logs are spammed with:

Jun  7 17:59:57 at-host-18 kernel: iser: iser_fast_reg_fmr: 
ib_fmr_pool_map_phys failed: -22
Jun  7 17:59:57 at-host-18 kernel: iser: iser_prepare_read_cmd: Failed 
to set up Data-IN RDMA
Jun  7 17:59:57 at-host-18 kernel: iser: iser_send_command: conn 
ffff88239eaf3b30 failed task->itt 127 err -22

Based on the error I found that there was a bug in 4.5.x rc Kernels and 
therefore tried 4.4.71, 4.9.31 and 4.9.27 but all of them have exactly 
the same behaviour.

I got HP DL160 (SE316) with QLogic Infiniband Adapters connected via a 
HP Voltair switch.

ibnetdiag says everything is fine.
Network via ib0 works without issues (can mount the iSCSI via that).
rdma_server and rdma_client both are successful (end 0)

ib_read_bw 10.0.13.3
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                     RDMA_Read BW Test
  Dual-port       : OFF		Device         : qib0
  Number of qps   : 1		Transport type : IB
  Connection type : RC		Using SRQ      : OFF
  TX depth        : 128
  CQ Moderation   : 100
  Mtu             : 4096[B]
  Link type       : IB
  Outstand reads  : 16
  rdma_cm QPs	 : OFF
  Data ex. method : Ethernet
---------------------------------------------------------------------------------------
  local address: LID 0x04 QPN 0x0017 PSN 0xd20eae OUT 0x10 RKey 
0x2d2d300 VAddr 0x007f892ea78000
  remote address: LID 0x13 QPN 0x0d7d PSN 0x66ee4e OUT 0x10 RKey 
0x070800 VAddr 0x007fcf485ee000
---------------------------------------------------------------------------------------
  #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   
MsgRate[Mpps]
  65536      1000             3256.60            3156.37		   0.050502
---------------------------------------------------------------------------------------

ib_send_bw 10.0.13.3
---------------------------------------------------------------------------------------
                     Send BW Test
  Dual-port       : OFF		Device         : qib0
  Number of qps   : 1		Transport type : IB
  Connection type : RC		Using SRQ      : OFF
  TX depth        : 128
  CQ Moderation   : 100
  Mtu             : 4096[B]
  Link type       : IB
  Max inline data : 0[B]
  rdma_cm QPs	 : OFF
  Data ex. method : Ethernet
---------------------------------------------------------------------------------------
  local address: LID 0x04 QPN 0x0019 PSN 0xae4e4c
  remote address: LID 0x13 QPN 0x0d89 PSN 0xd25041
---------------------------------------------------------------------------------------
  #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   
MsgRate[Mpps]
  65536      1000             3232.96            3232.92		   0.051727
---------------------------------------------------------------------------------------

ib_read and ib_send also look fine. (Throughput as expected with 3.2 
GB/s - 40 Gbit/s QDR)

Also srpt is working, but I need iSCSI.




Does anyone know what the issue could be, or how I can better analyze 
what's happening there?

Thanks
BR
Thomas
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER Connection via LIO not working

[Robert LeBlanc]

Can you post the config for lio and the iscsiadm commands you are
using? Have you deleted the record from the discoverydb and
rediscovered? You need to be sure the enable_iser boolean is set on
the target and that the modules (ib_isert and ib_iser) are loaded.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Wed, Jun 7, 2017 at 12:15 PM, Thomas Rosenstein
<thomas.rosenstein-+32LHaKS/h5VtLMLtJdGzA@public.gmane.org> wrote:
> Hello,
>
> i have setup a lio with 1 LUN on kernel 4.9.11, works fine via iSCSI, but
> when switching to iSER the message logs are spammed with:
>
> Jun  7 17:59:57 at-host-18 kernel: iser: iser_fast_reg_fmr:
> ib_fmr_pool_map_phys failed: -22
> Jun  7 17:59:57 at-host-18 kernel: iser: iser_prepare_read_cmd: Failed to
> set up Data-IN RDMA
> Jun  7 17:59:57 at-host-18 kernel: iser: iser_send_command: conn
> ffff88239eaf3b30 failed task->itt 127 err -22
>
> Based on the error I found that there was a bug in 4.5.x rc Kernels and
> therefore tried 4.4.71, 4.9.31 and 4.9.27 but all of them have exactly the
> same behaviour.
>
> I got HP DL160 (SE316) with QLogic Infiniband Adapters connected via a HP
> Voltair switch.
>
> ibnetdiag says everything is fine.
> Network via ib0 works without issues (can mount the iSCSI via that).
> rdma_server and rdma_client both are successful (end 0)
>
> ib_read_bw 10.0.13.3
> ---------------------------------------------------------------------------------------
> Device not recognized to implement inline feature. Disabling it
> ---------------------------------------------------------------------------------------
>                     RDMA_Read BW Test
>  Dual-port       : OFF          Device         : qib0
>  Number of qps   : 1            Transport type : IB
>  Connection type : RC           Using SRQ      : OFF
>  TX depth        : 128
>  CQ Moderation   : 100
>  Mtu             : 4096[B]
>  Link type       : IB
>  Outstand reads  : 16
>  rdma_cm QPs     : OFF
>  Data ex. method : Ethernet
> ---------------------------------------------------------------------------------------
>  local address: LID 0x04 QPN 0x0017 PSN 0xd20eae OUT 0x10 RKey 0x2d2d300
> VAddr 0x007f892ea78000
>  remote address: LID 0x13 QPN 0x0d7d PSN 0x66ee4e OUT 0x10 RKey 0x070800
> VAddr 0x007fcf485ee000
> ---------------------------------------------------------------------------------------
>  #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]
> MsgRate[Mpps]
>  65536      1000             3256.60            3156.37            0.050502
> ---------------------------------------------------------------------------------------
>
> ib_send_bw 10.0.13.3
> ---------------------------------------------------------------------------------------
>                     Send BW Test
>  Dual-port       : OFF          Device         : qib0
>  Number of qps   : 1            Transport type : IB
>  Connection type : RC           Using SRQ      : OFF
>  TX depth        : 128
>  CQ Moderation   : 100
>  Mtu             : 4096[B]
>  Link type       : IB
>  Max inline data : 0[B]
>  rdma_cm QPs     : OFF
>  Data ex. method : Ethernet
> ---------------------------------------------------------------------------------------
>  local address: LID 0x04 QPN 0x0019 PSN 0xae4e4c
>  remote address: LID 0x13 QPN 0x0d89 PSN 0xd25041
> ---------------------------------------------------------------------------------------
>  #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]
> MsgRate[Mpps]
>  65536      1000             3232.96            3232.92            0.051727
> ---------------------------------------------------------------------------------------
>
> ib_read and ib_send also look fine. (Throughput as expected with 3.2 GB/s -
> 40 Gbit/s QDR)
>
> Also srpt is working, but I need iSCSI.
>
>
>
>
> Does anyone know what the issue could be, or how I can better analyze what's
> happening there?
>
> Thanks
> BR
> Thomas
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER Connection via LIO not working

[Thomas Rosenstein]

[-- Attachment #1: Type: text/plain, Size: 7419 bytes --]

Hi,

lio config (extract - .json file attached)

o- iscsi 
........................................................................................... 
[1-way disc auth, Targets: 1]
   | o- iqn.2017-01.at.cf-it.at-storage-01 
................................................................................ 
[TPGs: 1]
   |   o- tpg1 
.......................................................................................... 
[no-gen-acls, auth per-acl]
   |     o- acls 
.......................................................................................................... 
[ACLs: 1]
   |     | o- iqn.2017-01.at.cf-it.kubernetes.db-1 
..................................................... [1-way auth, 
Mapped LUNs: 1]
   |     |   o- mapped_lun0 
...................................................................... 
[lun0 fileio/kubernetes_db_1 (rw)]
   |     o- luns 
.......................................................................................................... 
[LUNs: 1]
   |     | o- lun0 .......................................... 
[fileio/kubernetes_db_1 (/data/kubernetes_storage/kubernetes_db_1.img)]
   |     o- portals 
.................................................................................................... 
[Portals: 1]
   |       o- 0.0.0.0:3260 
................................................................................................... 
[iser]

at-storage-01 modules:

ib_isert               49152  4
iscsi_target_mod      294912  8 ib_isert
ib_iser                49152  0
libiscsi               57344  1 ib_iser
scsi_transport_iscsi    98304  2 ib_iser,libiscsi
target_core_mod       352256  43 
iscsi_target_mod,ib_isert,target_core_iblock,ib_srpt,target_core_user,target_core_file,target_core_pscsi
rdma_cm                53248  4 ib_iser,ib_isert,rpcrdma,rdma_ucm
ib_core               208896  15 
ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,ib_srpt,ib_ucm,rdmavt,ib_qib,rdma_ucm

at-host-18 modules:

ib_isert               49152  0
iscsi_target_mod      299008  1 ib_isert
ib_iser                49152  0
libiscsi               57344  1 ib_iser
scsi_transport_iscsi    98304  3 ib_iser,libiscsi
target_core_mod       352256  3 iscsi_target_mod,ib_isert,ib_srpt
rdma_cm                53248  4 ib_iser,ib_isert,rpcrdma,rdma_ucm
ib_core               208896  15 
ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,ib_srpt,ib_ucm,rdmavt,ib_qib,rdma_ucm


Commands to initialize iscsiadm:

iscsiadm -m discovery -t st -p 10.0.13.3:3260 -I iser

iscsiadm -m node -P 1
Target: iqn.2017-01.at.cf-it.at-storage-01
	Portal: 10.0.13.3:3260,1
		Iface Name: iser

iscsiadm -m node -l

->> creates the logs


@deleted and recreated, multiple times
ib_isert and ib_iser are loaded on both servers

BR
Thomas

On 7 Jun 2017, at 20:35, Robert LeBlanc wrote:

> Can you post the config for lio and the iscsiadm commands you are
> using? Have you deleted the record from the discoverydb and
> rediscovered? You need to be sure the enable_iser boolean is set on
> the target and that the modules (ib_isert and ib_iser) are loaded.
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>
>
> On Wed, Jun 7, 2017 at 12:15 PM, Thomas Rosenstein
> <thomas.rosenstein-+32LHaKS/h5VtLMLtJdGzA@public.gmane.org> wrote:
>> Hello,
>>
>> i have setup a lio with 1 LUN on kernel 4.9.11, works fine via iSCSI, 
>> but
>> when switching to iSER the message logs are spammed with:
>>
>> Jun  7 17:59:57 at-host-18 kernel: iser: iser_fast_reg_fmr:
>> ib_fmr_pool_map_phys failed: -22
>> Jun  7 17:59:57 at-host-18 kernel: iser: iser_prepare_read_cmd: 
>> Failed to
>> set up Data-IN RDMA
>> Jun  7 17:59:57 at-host-18 kernel: iser: iser_send_command: conn
>> ffff88239eaf3b30 failed task->itt 127 err -22
>>
>> Based on the error I found that there was a bug in 4.5.x rc Kernels 
>> and
>> therefore tried 4.4.71, 4.9.31 and 4.9.27 but all of them have 
>> exactly the
>> same behaviour.
>>
>> I got HP DL160 (SE316) with QLogic Infiniband Adapters connected via 
>> a HP
>> Voltair switch.
>>
>> ibnetdiag says everything is fine.
>> Network via ib0 works without issues (can mount the iSCSI via that).
>> rdma_server and rdma_client both are successful (end 0)
>>
>> ib_read_bw 10.0.13.3
>> ---------------------------------------------------------------------------------------
>> Device not recognized to implement inline feature. Disabling it
>> ---------------------------------------------------------------------------------------
>>                     RDMA_Read BW Test
>>  Dual-port       : OFF          Device         : qib0
>>  Number of qps   : 1            Transport type : IB
>>  Connection type : RC           Using SRQ      : OFF
>>  TX depth        : 128
>>  CQ Moderation   : 100
>>  Mtu             : 4096[B]
>>  Link type       : IB
>>  Outstand reads  : 16
>>  rdma_cm QPs     : OFF
>>  Data ex. method : Ethernet
>> ---------------------------------------------------------------------------------------
>>  local address: LID 0x04 QPN 0x0017 PSN 0xd20eae OUT 0x10 RKey 
>> 0x2d2d300
>> VAddr 0x007f892ea78000
>>  remote address: LID 0x13 QPN 0x0d7d PSN 0x66ee4e OUT 0x10 RKey 
>> 0x070800
>> VAddr 0x007fcf485ee000
>> ---------------------------------------------------------------------------------------
>>  #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]
>> MsgRate[Mpps]
>>  65536      1000             3256.60            3156.37            
>> 0.050502
>> ---------------------------------------------------------------------------------------
>>
>> ib_send_bw 10.0.13.3
>> ---------------------------------------------------------------------------------------
>>                     Send BW Test
>>  Dual-port       : OFF          Device         : qib0
>>  Number of qps   : 1            Transport type : IB
>>  Connection type : RC           Using SRQ      : OFF
>>  TX depth        : 128
>>  CQ Moderation   : 100
>>  Mtu             : 4096[B]
>>  Link type       : IB
>>  Max inline data : 0[B]
>>  rdma_cm QPs     : OFF
>>  Data ex. method : Ethernet
>> ---------------------------------------------------------------------------------------
>>  local address: LID 0x04 QPN 0x0019 PSN 0xae4e4c
>>  remote address: LID 0x13 QPN 0x0d89 PSN 0xd25041
>> ---------------------------------------------------------------------------------------
>>  #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]
>> MsgRate[Mpps]
>>  65536      1000             3232.96            3232.92            
>> 0.051727
>> ---------------------------------------------------------------------------------------
>>
>> ib_read and ib_send also look fine. (Throughput as expected with 3.2 
>> GB/s -
>> 40 Gbit/s QDR)
>>
>> Also srpt is working, but I need iSCSI.
>>
>>
>>
>>
>> Does anyone know what the issue could be, or how I can better analyze 
>> what's
>> happening there?
>>
>> Thanks
>> BR
>> Thomas
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" 
>> in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: saveconfig.json --]
[-- Type: application/json, Size: 11096 bytes --]

Re: iSER Connection via LIO not working

[Robert LeBlanc]

On Wed, Jun 7, 2017 at 12:46 PM, Thomas Rosenstein
<thomas.rosenstein-+32LHaKS/h5VtLMLtJdGzA@public.gmane.org> wrote:
> Hi,
>
> lio config (extract - .json file attached)
>
> o- iscsi
> ...........................................................................................
> [1-way disc auth, Targets: 1]
>   | o- iqn.2017-01.at.cf-it.at-storage-01
> ................................................................................
> [TPGs: 1]
>   |   o- tpg1
> ..........................................................................................
> [no-gen-acls, auth per-acl]
>   |     o- acls
> ..........................................................................................................
> [ACLs: 1]
>   |     | o- iqn.2017-01.at.cf-it.kubernetes.db-1
> ..................................................... [1-way auth, Mapped
> LUNs: 1]
>   |     |   o- mapped_lun0
> ...................................................................... [lun0
> fileio/kubernetes_db_1 (rw)]
>   |     o- luns
> ..........................................................................................................
> [LUNs: 1]
>   |     | o- lun0 ..........................................
> [fileio/kubernetes_db_1 (/data/kubernetes_storage/kubernetes_db_1.img)]
>   |     o- portals
> ....................................................................................................
> [Portals: 1]
>   |       o- 0.0.0.0:3260
> ...................................................................................................
> [iser]
>
> at-storage-01 modules:
>
> ib_isert               49152  4
> iscsi_target_mod      294912  8 ib_isert
> ib_iser                49152  0
> libiscsi               57344  1 ib_iser
> scsi_transport_iscsi    98304  2 ib_iser,libiscsi
> target_core_mod       352256  43
> iscsi_target_mod,ib_isert,target_core_iblock,ib_srpt,target_core_user,target_core_file,target_core_pscsi
> rdma_cm                53248  4 ib_iser,ib_isert,rpcrdma,rdma_ucm
> ib_core               208896  15
> ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,ib_srpt,ib_ucm,rdmavt,ib_qib,rdma_ucm
>
> at-host-18 modules:
>
> ib_isert               49152  0
> iscsi_target_mod      299008  1 ib_isert
> ib_iser                49152  0
> libiscsi               57344  1 ib_iser
> scsi_transport_iscsi    98304  3 ib_iser,libiscsi
> target_core_mod       352256  3 iscsi_target_mod,ib_isert,ib_srpt
> rdma_cm                53248  4 ib_iser,ib_isert,rpcrdma,rdma_ucm
> ib_core               208896  15
> ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,ib_srpt,ib_ucm,rdmavt,ib_qib,rdma_ucm
>
>
> Commands to initialize iscsiadm:
>
> iscsiadm -m discovery -t st -p 10.0.13.3:3260 -I iser
>
> iscsiadm -m node -P 1
> Target: iqn.2017-01.at.cf-it.at-storage-01
>         Portal: 10.0.13.3:3260,1
>                 Iface Name: iser
>
> iscsiadm -m node -l
>
> ->> creates the logs
>
>
> @deleted and recreated, multiple times
> ib_isert and ib_iser are loaded on both servers
>
> BR
> Thomas

The config looks good to me. I did have a similar problem with a RoCE
card where I could not get iSER working although RDMA did work.
Looking at the code, it seems that maybe the QLogic driver is not
properly allocating some memory that iSER is trying to use. I'm not an
expert in this so I could be completely off base here, just trying to
help out.

If you are adventerious, you can add some print statements in
drivers/infiniband/core/fmr_pool.c in the ib_fmr_pool_map_phys
function to see what may be going on. I think you are hitting this
line:

        if (list_len < 1 || list_len > pool->max_pages)
                return ERR_PTR(-EINVAL);

I would check the values of list_len and pool->max_pages before that.

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER Connection via LIO not working

[Thomas Rosenstein]

Okay, just did that:

printk(KERN_WARNING PFX "Debug pool %d - %d\n", list_len, 
pool->max_pages);

result is:

fmr_pool: Debug pool 1 - 0

means that the max_pages is set to 0


any idea how why and how to solve that?

>
> On 7 Jun 2017, at 21:05, Robert LeBlanc wrote:
>
>> On Wed, Jun 7, 2017 at 12:46 PM, Thomas Rosenstein
>> <thomas.rosenstein-+32LHaKS/h5VtLMLtJdGzA@public.gmane.org> wrote:
>>> Hi,
>>>
>>> lio config (extract - .json file attached)
>>>
>>> o- iscsi
>>> ...........................................................................................
>>> [1-way disc auth, Targets: 1]
>>>   | o- iqn.2017-01.at.cf-it.at-storage-01
>>> ................................................................................
>>> [TPGs: 1]
>>>   |   o- tpg1
>>> ..........................................................................................
>>> [no-gen-acls, auth per-acl]
>>>   |     o- acls
>>> ..........................................................................................................
>>> [ACLs: 1]
>>>   |     | o- iqn.2017-01.at.cf-it.kubernetes.db-1
>>> ..................................................... [1-way auth, 
>>> Mapped
>>> LUNs: 1]
>>>   |     |   o- mapped_lun0
>>> ...................................................................... 
>>> [lun0
>>> fileio/kubernetes_db_1 (rw)]
>>>   |     o- luns
>>> ..........................................................................................................
>>> [LUNs: 1]
>>>   |     | o- lun0 ..........................................
>>> [fileio/kubernetes_db_1 
>>> (/data/kubernetes_storage/kubernetes_db_1.img)]
>>>   |     o- portals
>>> ....................................................................................................
>>> [Portals: 1]
>>>   |       o- 0.0.0.0:3260
>>> ...................................................................................................
>>> [iser]
>>>
>>> at-storage-01 modules:
>>>
>>> ib_isert               49152  4
>>> iscsi_target_mod      294912  8 ib_isert
>>> ib_iser                49152  0
>>> libiscsi               57344  1 ib_iser
>>> scsi_transport_iscsi    98304  2 ib_iser,libiscsi
>>> target_core_mod       352256  43
>>> iscsi_target_mod,ib_isert,target_core_iblock,ib_srpt,target_core_user,target_core_file,target_core_pscsi
>>> rdma_cm                53248  4 ib_iser,ib_isert,rpcrdma,rdma_ucm
>>> ib_core               208896  15
>>> ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,ib_srpt,ib_ucm,rdmavt,ib_qib,rdma_ucm
>>>
>>> at-host-18 modules:
>>>
>>> ib_isert               49152  0
>>> iscsi_target_mod      299008  1 ib_isert
>>> ib_iser                49152  0
>>> libiscsi               57344  1 ib_iser
>>> scsi_transport_iscsi    98304  3 ib_iser,libiscsi
>>> target_core_mod       352256  3 iscsi_target_mod,ib_isert,ib_srpt
>>> rdma_cm                53248  4 ib_iser,ib_isert,rpcrdma,rdma_ucm
>>> ib_core               208896  15
>>> ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,ib_srpt,ib_ucm,rdmavt,ib_qib,rdma_ucm
>>>
>>>
>>> Commands to initialize iscsiadm:
>>>
>>> iscsiadm -m discovery -t st -p 10.0.13.3:3260 -I iser
>>>
>>> iscsiadm -m node -P 1
>>> Target: iqn.2017-01.at.cf-it.at-storage-01
>>>         Portal: 10.0.13.3:3260,1
>>>                 Iface Name: iser
>>>
>>> iscsiadm -m node -l
>>>
>>> ->> creates the logs
>>>
>>>
>>> @deleted and recreated, multiple times
>>> ib_isert and ib_iser are loaded on both servers
>>>
>>> BR
>>> Thomas
>>
>> The config looks good to me. I did have a similar problem with a RoCE
>> card where I could not get iSER working although RDMA did work.
>> Looking at the code, it seems that maybe the QLogic driver is not
>> properly allocating some memory that iSER is trying to use. I'm not 
>> an
>> expert in this so I could be completely off base here, just trying to
>> help out.
>>
>> If you are adventerious, you can add some print statements in
>> drivers/infiniband/core/fmr_pool.c in the ib_fmr_pool_map_phys
>> function to see what may be going on. I think you are hitting this
>> line:
>>
>>         if (list_len < 1 || list_len > pool->max_pages)
>>                 return ERR_PTR(-EINVAL);
>>
>> I would check the values of list_len and pool->max_pages before that.
>>
>> ----------------
>> Robert LeBlanc
>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER Connection via LIO not working

[Robert LeBlanc]

On Wed, Jun 7, 2017 at 1:26 PM, Thomas Rosenstein
<thomas.rosenstein-+32LHaKS/h5VtLMLtJdGzA@public.gmane.org> wrote:
> Okay, just did that:
>
> printk(KERN_WARNING PFX "Debug pool %d - %d\n", list_len, pool->max_pages);
>
> result is:
>
> fmr_pool: Debug pool 1 - 0
>
> means that the max_pages is set to 0
>
> any idea how why and how to solve that?
>

Again, this is out of my wheelhouse and I'm having trouble tracing all
the callbacks (I usually have to do a ton of print statements to
figure out the callbacks). I do think that the pool_handle is being
set further down in drivers/infiniband/ulp/iser/iser_memory.c in the
iser_reg_rdma_mem() funtion as part of the "struct iser_fr_desc *desc"
component. I think it is probably being set in this code:

        if (!use_dma_key) {
                desc = device->reg_ops->reg_desc_get(ib_conn);
                reg->mem_h = desc;
        }

Now it could be set by reg_desc_get (which I think is an iSER
callback) or more likely as part of ib_conn which I think the driver
sets up. Hopefully someone smarter than I can chime in and give you
some help as to why the pool->max_pages is zero.

Good luck!

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Wed, Jun 7, 2017 at 1:26 PM, Thomas Rosenstein
<thomas.rosenstein-+32LHaKS/h5VtLMLtJdGzA@public.gmane.org> wrote:
> Okay, just did that:
>
> printk(KERN_WARNING PFX "Debug pool %d - %d\n", list_len, pool->max_pages);
>
> result is:
>
> fmr_pool: Debug pool 1 - 0
>
> means that the max_pages is set to 0
>
> any idea how why and how to solve that?
>
> On 7 Jun 2017, at 21:05, Robert LeBlanc wrote:
>
> On Wed, Jun 7, 2017 at 12:46 PM, Thomas Rosenstein
> <thomas.rosenstein-+32LHaKS/h5VtLMLtJdGzA@public.gmane.org> wrote:
>
> Hi,
>
> lio config (extract - .json file attached)
>
> o- iscsi
> ...........................................................................................
> [1-way disc auth, Targets: 1]
> | o- iqn.2017-01.at.cf-it.at-storage-01
> ................................................................................
> [TPGs: 1]
> | o- tpg1
> ..........................................................................................
> [no-gen-acls, auth per-acl]
> | o- acls
> ..........................................................................................................
> [ACLs: 1]
> | | o- iqn.2017-01.at.cf-it.kubernetes.db-1
> ..................................................... [1-way auth, Mapped
> LUNs: 1]
> | | o- mapped_lun0
> ...................................................................... [lun0
> fileio/kubernetes_db_1 (rw)]
> | o- luns
> ..........................................................................................................
> [LUNs: 1]
> | | o- lun0 ..........................................
> [fileio/kubernetes_db_1 (/data/kubernetes_storage/kubernetes_db_1.img)]
> | o- portals
> ....................................................................................................
> [Portals: 1]
> | o- 0.0.0.0:3260
> ...................................................................................................
> [iser]
>
> at-storage-01 modules:
>
> ib_isert 49152 4
> iscsi_target_mod 294912 8 ib_isert
> ib_iser 49152 0
> libiscsi 57344 1 ib_iser
> scsi_transport_iscsi 98304 2 ib_iser,libiscsi
> target_core_mod 352256 43
> iscsi_target_mod,ib_isert,target_core_iblock,ib_srpt,target_core_user,target_core_file,target_core_pscsi
> rdma_cm 53248 4 ib_iser,ib_isert,rpcrdma,rdma_ucm
> ib_core 208896 15
> ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,ib_srpt,ib_ucm,rdmavt,ib_qib,rdma_ucm
>
> at-host-18 modules:
>
> ib_isert 49152 0
> iscsi_target_mod 299008 1 ib_isert
> ib_iser 49152 0
> libiscsi 57344 1 ib_iser
> scsi_transport_iscsi 98304 3 ib_iser,libiscsi
> target_core_mod 352256 3 iscsi_target_mod,ib_isert,ib_srpt
> rdma_cm 53248 4 ib_iser,ib_isert,rpcrdma,rdma_ucm
> ib_core 208896 15
> ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,ib_srpt,ib_ucm,rdmavt,ib_qib,rdma_ucm
>
>
> Commands to initialize iscsiadm:
>
> iscsiadm -m discovery -t st -p 10.0.13.3:3260 -I iser
>
> iscsiadm -m node -P 1
> Target: iqn.2017-01.at.cf-it.at-storage-01
> Portal: 10.0.13.3:3260,1
> Iface Name: iser
>
> iscsiadm -m node -l
>
> ->> creates the logs
>
>
> @deleted and recreated, multiple times
> ib_isert and ib_iser are loaded on both servers
>
> BR
> Thomas
>
> The config looks good to me. I did have a similar problem with a RoCE
> card where I could not get iSER working although RDMA did work.
> Looking at the code, it seems that maybe the QLogic driver is not
> properly allocating some memory that iSER is trying to use. I'm not an
> expert in this so I could be completely off base here, just trying to
> help out.
>
> If you are adventerious, you can add some print statements in
> drivers/infiniband/core/fmr_pool.c in the ib_fmr_pool_map_phys
> function to see what may be going on. I think you are hitting this
> line:
>
> if (list_len < 1 || list_len > pool->max_pages)
> return ERR_PTR(-EINVAL);
>
> I would check the values of list_len and pool->max_pages before that.
>
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER Connection via LIO not working

[Thomas Rosenstein]

Hi,

after a bit of digging and adding about 100 logs I found the following 
commit which screws with the qib implementation.

https://gitlab-beta.engr.illinois.edu/ejclark2/linux/commit/df749cdc45d9f97cb0a5e6ceab80e2e00ee9bf85

If you check, qib never sets the max_fast_reg_page_list_len (I assume, 
haven't found it in the code, it's there for mellanox)

therefore in iser_calc_scsi_params, the scsi_max_sectors is always set 
to 0.

Maybe someone that has experience there could figure out what exactly is 
going on?

Adding

sup_sg_tablesize = ISCSI_ISER_MAX_SG_TABLESIZE;

directly under the sup_sg_tablesize calculation at least fixes the iser 
connection for QLogic Adapters (~800 - 900 MB/s direct IO vs. ~200 MB/s 
via iscsi and no more errors)

Regards

On 7 Jun 2017, at 21:46, Robert LeBlanc wrote:

> On Wed, Jun 7, 2017 at 1:26 PM, Thomas Rosenstein
> <thomas.rosenstein-+32LHaKS/h5VtLMLtJdGzA@public.gmane.org> wrote:
>> Okay, just did that:
>>
>> printk(KERN_WARNING PFX "Debug pool %d - %d\n", list_len, 
>> pool->max_pages);
>>
>> result is:
>>
>> fmr_pool: Debug pool 1 - 0
>>
>> means that the max_pages is set to 0
>>
>> any idea how why and how to solve that?
>>
>
> Again, this is out of my wheelhouse and I'm having trouble tracing all
> the callbacks (I usually have to do a ton of print statements to
> figure out the callbacks). I do think that the pool_handle is being
> set further down in drivers/infiniband/ulp/iser/iser_memory.c in the
> iser_reg_rdma_mem() funtion as part of the "struct iser_fr_desc *desc"
> component. I think it is probably being set in this code:
>
>         if (!use_dma_key) {
>                 desc = device->reg_ops->reg_desc_get(ib_conn);
>                 reg->mem_h = desc;
>         }
>
> Now it could be set by reg_desc_get (which I think is an iSER
> callback) or more likely as part of ib_conn which I think the driver
> sets up. Hopefully someone smarter than I can chime in and give you
> some help as to why the pool->max_pages is zero.
>
> Good luck!
>
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>
>
> On Wed, Jun 7, 2017 at 1:26 PM, Thomas Rosenstein
> <thomas.rosenstein-+32LHaKS/h5VtLMLtJdGzA@public.gmane.org> wrote:
>> Okay, just did that:
>>
>> printk(KERN_WARNING PFX "Debug pool %d - %d\n", list_len, 
>> pool->max_pages);
>>
>> result is:
>>
>> fmr_pool: Debug pool 1 - 0
>>
>> means that the max_pages is set to 0
>>
>> any idea how why and how to solve that?
>>
>> On 7 Jun 2017, at 21:05, Robert LeBlanc wrote:
>>
>> On Wed, Jun 7, 2017 at 12:46 PM, Thomas Rosenstein
>> <thomas.rosenstein-+32LHaKS/h5VtLMLtJdGzA@public.gmane.org> wrote:
>>
>> Hi,
>>
>> lio config (extract - .json file attached)
>>
>> o- iscsi
>> ...........................................................................................
>> [1-way disc auth, Targets: 1]
>> | o- iqn.2017-01.at.cf-it.at-storage-01
>> ................................................................................
>> [TPGs: 1]
>> | o- tpg1
>> ..........................................................................................
>> [no-gen-acls, auth per-acl]
>> | o- acls
>> ..........................................................................................................
>> [ACLs: 1]
>> | | o- iqn.2017-01.at.cf-it.kubernetes.db-1
>> ..................................................... [1-way auth, 
>> Mapped
>> LUNs: 1]
>> | | o- mapped_lun0
>> ...................................................................... 
>> [lun0
>> fileio/kubernetes_db_1 (rw)]
>> | o- luns
>> ..........................................................................................................
>> [LUNs: 1]
>> | | o- lun0 ..........................................
>> [fileio/kubernetes_db_1 
>> (/data/kubernetes_storage/kubernetes_db_1.img)]
>> | o- portals
>> ....................................................................................................
>> [Portals: 1]
>> | o- 0.0.0.0:3260
>> ...................................................................................................
>> [iser]
>>
>> at-storage-01 modules:
>>
>> ib_isert 49152 4
>> iscsi_target_mod 294912 8 ib_isert
>> ib_iser 49152 0
>> libiscsi 57344 1 ib_iser
>> scsi_transport_iscsi 98304 2 ib_iser,libiscsi
>> target_core_mod 352256 43
>> iscsi_target_mod,ib_isert,target_core_iblock,ib_srpt,target_core_user,target_core_file,target_core_pscsi
>> rdma_cm 53248 4 ib_iser,ib_isert,rpcrdma,rdma_ucm
>> ib_core 208896 15
>> ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,ib_srpt,ib_ucm,rdmavt,ib_qib,rdma_ucm
>>
>> at-host-18 modules:
>>
>> ib_isert 49152 0
>> iscsi_target_mod 299008 1 ib_isert
>> ib_iser 49152 0
>> libiscsi 57344 1 ib_iser
>> scsi_transport_iscsi 98304 3 ib_iser,libiscsi
>> target_core_mod 352256 3 iscsi_target_mod,ib_isert,ib_srpt
>> rdma_cm 53248 4 ib_iser,ib_isert,rpcrdma,rdma_ucm
>> ib_core 208896 15
>> ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,ib_srpt,ib_ucm,rdmavt,ib_qib,rdma_ucm
>>
>>
>> Commands to initialize iscsiadm:
>>
>> iscsiadm -m discovery -t st -p 10.0.13.3:3260 -I iser
>>
>> iscsiadm -m node -P 1
>> Target: iqn.2017-01.at.cf-it.at-storage-01
>> Portal: 10.0.13.3:3260,1
>> Iface Name: iser
>>
>> iscsiadm -m node -l
>>
>> ->> creates the logs
>>
>>
>> @deleted and recreated, multiple times
>> ib_isert and ib_iser are loaded on both servers
>>
>> BR
>> Thomas
>>
>> The config looks good to me. I did have a similar problem with a RoCE
>> card where I could not get iSER working although RDMA did work.
>> Looking at the code, it seems that maybe the QLogic driver is not
>> properly allocating some memory that iSER is trying to use. I'm not 
>> an
>> expert in this so I could be completely off base here, just trying to
>> help out.
>>
>> If you are adventerious, you can add some print statements in
>> drivers/infiniband/core/fmr_pool.c in the ib_fmr_pool_map_phys
>> function to see what may be going on. I think you are hitting this
>> line:
>>
>> if (list_len < 1 || list_len > pool->max_pages)
>> return ERR_PTR(-EINVAL);
>>
>> I would check the values of list_len and pool->max_pages before that.
>>
>> ----------------
>> Robert LeBlanc
>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: iSER Connection via LIO not working

[Marciniszyn, Mike]

> Subject: Re: iSER Connection via LIO not working
> 
> Hi,
> 
> after a bit of digging and adding about 100 logs I found the following
> commit which screws with the qib implementation.
> 
> https://gitlab-
> beta.engr.illinois.edu/ejclark2/linux/commit/df749cdc45d9f97cb0a5e6ceab80
> e2e00ee9bf85
> 
> If you check, qib never sets the max_fast_reg_page_list_len (I assume,
> haven't found it in the code, it's there for mellanox)
> 
> therefore in iser_calc_scsi_params, the scsi_max_sectors is always set
> to 0.
> 
> Maybe someone that has experience there could figure out what exactly is
> going on?
> 
> Adding
> 
> sup_sg_tablesize = ISCSI_ISER_MAX_SG_TABLESIZE;
> 
> directly under the sup_sg_tablesize calculation at least fixes the iser
> connection for QLogic Adapters (~800 - 900 MB/s direct IO vs. ~200 MB/s
> via iscsi and no more errors)
> 

Good job tracking this down.   Thanks!

static void
iser_calc_scsi_params(struct iser_conn *iser_conn,
                      unsigned int max_sectors)
{
        struct iser_device *device = iser_conn->ib_conn.device;
        unsigned short sg_tablesize, sup_sg_tablesize;

        sg_tablesize = DIV_ROUND_UP(max_sectors * 512, SIZE_4K);
        sup_sg_tablesize = min_t(unsigned, ISCSI_ISER_MAX_SG_TABLESIZE,
                                 device->ib_device->attrs.max_fast_reg_page_list_len);

        iser_conn->scsi_sg_tablesize = min(sg_tablesize, sup_sg_tablesize);
}

The bug is that device->ib_device->attrs.max_fast_reg_page_list_len should only be considered valid IFF IB_DEVICE_MEM_MGT_EXTENSIONS is set in
device->ib_device->device_cap_flags.  The assignment you suggest should be an else on an if test of the device_cap_flags bit. 

This appears to be the only storage ULP that reads this field exported from drivers.

Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER Connection via LIO not working

[Bart Van Assche]

On Thu, 2017-06-08 at 16:23 +0000, Marciniszyn, Mike wrote:
> This appears to be the only storage ULP that reads this field exported from drivers.

Hello Mike,

Are you sure of this? This is what grep tells me:

$ git grep -lw max_fast_reg_page_list_len | grep -Ev '^drivers/infiniband/hw/|^include/'
drivers/infiniband/core/rw.c
drivers/infiniband/sw/rxe/rxe.c
drivers/infiniband/ulp/iser/iser_verbs.c
drivers/infiniband/ulp/srp/ib_srp.c
drivers/nvme/host/rdma.c
net/sunrpc/xprtrdma/frwr_ops.c
net/sunrpc/xprtrdma/svc_rdma_transport.c

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: iSER Connection via LIO not working

[Marciniszyn, Mike]

> From: Bart Van Assche [mailto:Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org]
> Sent: Thursday, June 8, 2017 12:28 PM
> To: robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org; thomas.rosenstein-+32LHaKS/h5VtLMLtJdGzA@public.gmane.org;
> Marciniszyn, Mike <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: Re: iSER Connection via LIO not working
> 
> On Thu, 2017-06-08 at 16:23 +0000, Marciniszyn, Mike wrote:
> > This appears to be the only storage ULP that reads this field exported from
> drivers.
> 
> Hello Mike,
> 
> Are you sure of this? This is what grep tells me:
> 

I only checked within ulp.    Perhaps it is a wider issue.

srp appears to only use the variable when use_fast_reg is set, which looks to be based on the device exported bit.

Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER Connection via LIO not working

[Thomas Rosenstein]

Hi

I ran into a second bug, but I'm not sure where to check that:

[150874.509144] iSCSI Target < 0
[150874.509146] iSCSI Login negotiation failed.
[150874.525818] Moving to TARG_CONN_STATE_FREE.
[150876.307247] Added timeout timer to iSCSI login request for 15 
seconds.
[150876.307250] Moving to TARG_CONN_STATE_XPT_UP.
[150877.307358] Received iSCSI login request from 10.0.13.23:36088 on 
IB/iSER Network Portal 10.0.13.3:3260
[150877.307360] Moving to TARG_CONN_STATE_IN_LOGIN.

the iSCSI Target < 0 is from me and is placed at

  rc = iscsi_target_locate_portal(np, conn, login);

  if (rc < 0) {
      pr_debug("iSCSI Target < 0");


I added more logs there, the issue is, it's only happening after a few 
hours with new connections!
Anyone experienced that, or has an idea what the issue could be?

BTW logging in the iSCSI Target and Portal needs to be improved!

BR
Thomas

On 8 Jun 2017, at 18:35, Marciniszyn, Mike wrote:

>> From: Bart Van Assche [mailto:Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org]
>> Sent: Thursday, June 8, 2017 12:28 PM
>> To: robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org; thomas.rosenstein-+32LHaKS/h5VtLMLtJdGzA@public.gmane.org;
>> Marciniszyn, Mike <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> Subject: Re: iSER Connection via LIO not working
>>
>> On Thu, 2017-06-08 at 16:23 +0000, Marciniszyn, Mike wrote:
>>> This appears to be the only storage ULP that reads this field 
>>> exported from
>> drivers.
>>
>> Hello Mike,
>>
>> Are you sure of this? This is what grep tells me:
>>
>
> I only checked within ulp.    Perhaps it is a wider issue.
>
> srp appears to only use the variable when use_fast_reg is set, which 
> looks to be based on the device exported bit.
>
> Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER Connection via LIO not working

[Sagi Grimberg]

> Good job tracking this down.   Thanks!
> 
> static void
> iser_calc_scsi_params(struct iser_conn *iser_conn,
>                        unsigned int max_sectors)
> {
>          struct iser_device *device = iser_conn->ib_conn.device;
>          unsigned short sg_tablesize, sup_sg_tablesize;
> 
>          sg_tablesize = DIV_ROUND_UP(max_sectors * 512, SIZE_4K);
>          sup_sg_tablesize = min_t(unsigned, ISCSI_ISER_MAX_SG_TABLESIZE,
>                                   device->ib_device->attrs.max_fast_reg_page_list_len);
> 
>          iser_conn->scsi_sg_tablesize = min(sg_tablesize, sup_sg_tablesize);
> }
> 
> The bug is that device->ib_device->attrs.max_fast_reg_page_list_len should only be considered valid IFF IB_DEVICE_MEM_MGT_EXTENSIONS is set in
> device->ib_device->device_cap_flags.  The assignment you suggest should be an else on an if test of the device_cap_flags bit.

Hi Guys,

Good job tracking it down!

I think you are the first one to test iSER on top of qib.

Mike, What other variable do we need to look at to determine the
maximum registered size per fmr? I was under the impression that
max_fast_reg_page_list_len is not directly related to
IB_DEVICE_MEM_MGT_EXTENSIONS as fmrs also accepts page lists.

Anyway,
Can we please please please add proper support for wr based memory
registrations in qib? I have a patch piped for removing FMRs from iser
altogether, now that this came up I obviously can't send it...

Looking at the code, looks like qib inherits rdmavt memory registration
with the new api, the only missing piece afaict is the remote invalidate
and qib can turn on IB_DEVICE_MEM_MGT_EXTENSIONS.

Can something like the (untested) below work:
--
diff --git a/drivers/infiniband/hw/qib/qib_rc.c 
b/drivers/infiniband/hw/qib/qib_rc.c
index 4ddbcac5eabe..2bdfad6ecc11 100644
--- a/drivers/infiniband/hw/qib/qib_rc.c
+++ b/drivers/infiniband/hw/qib/qib_rc.c
@@ -1861,6 +1861,7 @@ void qib_rc_rcv(struct qib_ctxtdata *rcd, struct 
ib_header *hdr,

         case OP(SEND_ONLY):
         case OP(SEND_ONLY_WITH_IMMEDIATE):
+       case OP(SEND_ONLY_WITH_INVALIDATE):
                 ret = qib_get_rwqe(qp, 0);
                 if (ret < 0)
                         goto nack_op_err;
@@ -1871,6 +1872,13 @@ void qib_rc_rcv(struct qib_ctxtdata *rcd, struct 
ib_header *hdr,
                         goto no_immediate_data;
                 /* FALLTHROUGH for SEND_ONLY_WITH_IMMEDIATE */
         case OP(SEND_LAST_WITH_IMMEDIATE):
+       case OP(SEND_LAST_WITH_INVALIDATE):
+               if (opcode == OP(SEND_LAST_WITH_INVALIDATE)) {
+                       wc.ex.invalidate_rkey = 
be32_to_cpu(ohdr->u.imm_data);
+                       rvt_invalidate_rkey(qp, wc.ex.invalidate_rkey);
+                       wc.wc_flags = IB_WC_WITH_INVALIDATE;
+                       goto send_last;
+               }
  send_last_imm:
                 wc.ex.imm_data = ohdr->u.imm_data;
                 hdrsize += 4;
--
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: iSER Connection via LIO not working

[Marciniszyn, Mike]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 2962 bytes --]

> 
> > Good job tracking this down.   Thanks!
> >
> > static void
> > iser_calc_scsi_params(struct iser_conn *iser_conn,
> >                        unsigned int max_sectors)
> > {
> >          struct iser_device *device = iser_conn->ib_conn.device;
> >          unsigned short sg_tablesize, sup_sg_tablesize;
> >
> >          sg_tablesize = DIV_ROUND_UP(max_sectors * 512, SIZE_4K);
> >          sup_sg_tablesize = min_t(unsigned, ISCSI_ISER_MAX_SG_TABLESIZE,
> >                                   device->ib_device->attrs.max_fast_reg_page_list_len);
> >
> >          iser_conn->scsi_sg_tablesize = min(sg_tablesize, sup_sg_tablesize);
> > }
> >
> > The bug is that device->ib_device->attrs.max_fast_reg_page_list_len
> should only be considered valid IFF IB_DEVICE_MEM_MGT_EXTENSIONS is
> set in
> > device->ib_device->device_cap_flags.  The assignment you suggest should
> be an else on an if test of the device_cap_flags bit.
> 
> Hi Guys,
> 
> Good job tracking it down!
> 
> I think you are the first one to test iSER on top of qib.
> 
> Mike, What other variable do we need to look at to determine the
> maximum registered size per fmr? I was under the impression that
> max_fast_reg_page_list_len is not directly related to
> IB_DEVICE_MEM_MGT_EXTENSIONS as fmrs also accepts page lists.
> 

I'm getting ready to send a patch.

For fmrs, there is no such indication available other than having the ULPs use trial fmr creation calls.

For qib, the only limit is memory to create the list embedded in the MR.   For hfi1, the limit is UINT_MAX.

I suggest that we just handle the case of NOT seeing the bit by assuming ISCSI_ISER_MAX_SG_TABLE_SIZE.

An alternative is for non-extension drivers to assign this field as they see fit.   Some other drivers seem to have limits that will fail creation.

max_fast_reg_page_list_len is poorly named though.

It seems simpler just to fix iser...
 
> Anyway,
> Can we please please please add proper support for wr based memory
> registrations in qib? I have a patch piped for removing FMRs from iser
> altogether, now that this came up I obviously can't send it...
> 

This cannot be done without dropping support for older hardware.

qib variants (and perhaps other hardware) cannot deal with the receiving the send with invalidate packets.

> Looking at the code, looks like qib inherits rdmavt memory registration
> with the new api, the only missing piece afaict is the remote invalidate
> and qib can turn on IB_DEVICE_MEM_MGT_EXTENSIONS.

The code in the patch you suggested will never get executed because of the above hardware limitation.   The hardware will queue an error and drop any data.

rdmavt reacts to control tables to handle the subordinate driver extension support correctly.

Mike






N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±­ÙšŠ{ayº\x1dʇڙë,j\a­¢f£¢·hš‹»öì\x17/oSc¾™Ú³9˜uÀ¦æå‰È&jw¨®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿïêäz¹Þ–Šàþf£¢·hšˆ§~ˆmš