* RDMA_CM_EVENT_REJECTED
From: Steve Wise @ 2016-10-25 14:38 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
While working on the reject helper functions, I notice that some of the rdma_cm
kernel users have handlers for RDMA_CM_EVENT_REJECTED even for cm_ids that are
spawned from listening cm_ids as the result of a RDMA_CM_EVENT_CONNECT_REQUEST,
and that issue an rdma_accept() or rdma_reject() in response.
Q: is it possible for a REJECTED event to get posted to these cm_ids? It is not
for iWARP, but I'm not sure about IB. I'm guessing this is just due to
cut/paste process of adding event handlers, but I wanted to make sure.
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: [PATCH v2 perftest] Support for Chelsio T6 devices
From: Steve Wise @ 2016-10-25 14:32 UTC (permalink / raw)
To: 'Gil Rockah', 'Zohar Ben Aharon'
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <01db01d22407$38da0930$aa8e1b90$@opengridcomputing.com>
Hey guys,
Has this patch been integrated yet? Also, where is the official upstream
perftest git repo now?
Thanks,
Steve.
> -----Original Message-----
> From: Steve Wise [mailto:swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org]
> Sent: Tuesday, October 11, 2016 4:34 PM
> To: 'Gil Rockah'; 'Zohar Ben Aharon'
> Subject: RE: [PATCH v2 perftest] Support for Chelsio T6 devices
>
>
> > -----Original Message-----
> > From: Gil Rockah [mailto:gilr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org]
> > Sent: Monday, September 26, 2016 2:01 AM
> > To: Zohar Ben Aharon; swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org
> > Subject: FW: [PATCH v2 perftest] Support for Chelsio T6 devices
> >
> > Hi Steve,
> > Thanks for the patch.
> > Please notice that Zohar is the new owner of perftest.
> >
> > Thanks,
> > Gil
> >
>
> Hey Zohar,
>
> Where is the formal perftest git repo now?
>
> Thanks,
>
> Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: NVMeoF Linux GIT repo
From: Robert Randall @ 2016-10-25 14:06 UTC (permalink / raw)
To: Sagi Grimberg
Cc: Robert Randall (rrandall), Keith Busch,
linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Haggai Eran
In-Reply-To: <2aeadd6a-0f5c-5c59-cafa-10116ccc91c0-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Any update on this from Haggi? Trying to understand if this is a
problem with our host driver or something on the Linux side.
Thanks,
Robert
On Fri, Oct 21, 2016 at 5:19 PM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote:
> Hey Robert,
>
>> Sorry Keith, I'm back to the same question again. I've tried using the
>> released 4.8.2 kernel and I'm seeing errors in the Linux RDMA layer. Log
>> file is attached. My guess is this may have been fixed already but since
>> I'm not writing code on Linux it is difficult to keep up with which repo and
>> which branch I should be using.
>>
>> It reports a syndrome 5 which appears to mean "work request flush error".
>>
>> Setup is stable 4.8.2 kernel with Mellanox RoCE v2.
>>
>> So, where do I grab the latest and greatest code these days?
>
>
> So from a quick look at the log the FLUSH errors are
> just side effects. Once a queue-pair transitions to
> ERROR state it flushes all the pending work requests with
> a FLUSH syndrome, so we should look at the first error which
> is:
>
> mlx5_1:poll_soft_wc:647:(pid 3422): polled software generated completion on
> CQ 0x14
>
> This seems to come from the GSI QP completion emulation from
> Haggai (CC'd). CQ 0x14 is not nvmet-rdma completion queue (from
> the log it's 0x5d) so something went wrong but its does not
> seem to be nvmet-rdma's fault.
>
> Haggai, any tips for Robert?
>
> Log output:
> [ 12.588248] mlx5_core 0000:06:00.0 enp6s0f0: Link up
> [ 12.735116] mlx5_core 0000:06:00.1 enp6s0f1: Link up
> [ 16.490224] e1000e: enp8s0 NIC Link is Up 1000 Mbps Full Duplex, Flow
> Control: Rx/Tx
> [ 20.005394] input: PS/2 Generic Mouse as
> /devices/platform/i8042/serio1/input/input4
> [ 33.138312] random: crng init done
> [ 57.710309] (0000:06:00.1): E-Switch: disable SRIOV: active vports(1)
> mode(0)
> [ 57.713889] (0000:06:00.1): E-Switch: cleanup
> [ 58.399819] (0000:06:00.0): E-Switch: disable SRIOV: active vports(1)
> mode(0)
> [ 58.401660] (0000:06:00.0): E-Switch: cleanup
> [ 59.134997] mlx5_core 0000:06:00.0: firmware version: 12.16.1020
> [ 59.855855] (0000:06:00.0): E-Switch: Total vports 9, l2 table
> size(65536), per vport: max uc(1024) max mc(16384)
> [ 59.857209] mlx5_core 0000:06:00.1: firmware version: 12.16.1020
> [ 60.563522] (0000:06:00.1): E-Switch: Total vports 9, l2 table
> size(65536), per vport: max uc(1024) max mc(16384)
> [ 60.566269] mlx5_core 0000:06:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(1)
> RxCqeCmprss(0)
> [ 60.737262] mlx5_core 0000:06:00.0 enp6s0f0: renamed from eth0
> [ 60.737617] mlx5_core 0000:06:00.1: MLX5E: StrdRq(0) RqSz(1024) StrdSz(1)
> RxCqeCmprss(0)
> [ 61.038325] mlx5_core 0000:06:00.0 enp6s0f0: Link up
> [ 61.041446] mlx5_core 0000:06:00.1 enp6s0f1: renamed from eth0
> [ 61.047290] mlx5_ib: Mellanox Connect-IB Infiniband driver v2.2-1 (Feb
> 2014)
> [ 61.981595] mlx5_core 0000:06:00.1 enp6s0f1: Link up
> [ 63.159807] e1000e: enp8s0 NIC Link is Down
> [ 67.836775] e1000e: enp8s0 NIC Link is Up 1000 Mbps Full Duplex, Flow
> Control: Rx/Tx
> [ 74.745144] mlx5_core 0000:06:00.0 enp6s0f0: Link up
> [ 74.944733] nvmet: adding nsid 1 to subsystem ignite1
> [ 74.945148] nvmet: adding nsid 2 to subsystem ignite2
> [ 74.945510] nvmet: adding nsid 3 to subsystem ignite3
> [ 74.945864] nvmet: adding nsid 4 to subsystem ignite4
> [ 74.946747] nvmet_rdma: enabling port 2 (192.168.2.10:5150)
> micron@fmslnx0:~$ tail -f /var/log/syslog
> Oct 20 07:28:17 fmslnx0 systemd[1]: Reloaded OpenBSD Secure Shell server.
> Oct 20 07:28:17 fmslnx0 kernel: [ 74.745144] mlx5_core 0000:06:00.0
> enp6s0f0: Link up
> Oct 20 07:28:17 fmslnx0 systemd[1]: Reloading OpenBSD Secure Shell server.
> Oct 20 07:28:17 fmslnx0 systemd[1]: Reloaded OpenBSD Secure Shell server.
> Oct 20 07:28:17 fmslnx0 systemd[1]: Started Raise network interfaces.
> Oct 20 07:28:18 fmslnx0 kernel: [ 74.944733] nvmet: adding nsid 1 to
> subsystem ignite1
> Oct 20 07:28:18 fmslnx0 kernel: [ 74.945148] nvmet: adding nsid 2 to
> subsystem ignite2
> Oct 20 07:28:18 fmslnx0 kernel: [ 74.945510] nvmet: adding nsid 3 to
> subsystem ignite3
> Oct 20 07:28:18 fmslnx0 kernel: [ 74.945864] nvmet: adding nsid 4 to
> subsystem ignite4
> Oct 20 07:28:18 fmslnx0 kernel: [ 74.946747] nvmet_rdma: enabling port 2
> (192.168.2.10:5150)
> Oct 20 07:35:01 fmslnx0 CRON[3376]: (root) CMD (command -v debian-sa1 >
> /dev/null && debian-sa1 1 1)
> Oct 20 07:42:35 fmslnx0 systemd[1]: Starting Cleanup of Temporary
> Directories...
> Oct 20 07:42:35 fmslnx0 systemd-tmpfiles[3381]:
> [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log",
> ignoring.
> Oct 20 07:42:35 fmslnx0 systemd[1]: Started Cleanup of Temporary
> Directories.
> Oct 20 07:45:01 fmslnx0 CRON[3395]: (root) CMD (command -v debian-sa1 >
> /dev/null && debian-sa1 1 1)
> Oct 20 07:50:48 fmslnx0 dhclient[3245]: DHCPREQUEST of 10.113.22.90 on
> enp8s0 to 10.113.22.46 port 67 (xid=0x19ae475f)
> Oct 20 07:50:48 fmslnx0 dhclient[3245]: DHCPACK of 10.113.22.90 from
> 10.113.22.46
> Oct 20 07:50:48 fmslnx0 dhclient[3245]: bound to 10.113.22.90 -- renewal in
> 1615 seconds.
> Oct 20 07:55:01 fmslnx0 CRON[3412]: (root) CMD (command -v debian-sa1 >
> /dev/null && debian-sa1 1 1)
> Oct 20 08:05:01 fmslnx0 CRON[3418]: (root) CMD (command -v debian-sa1 >
> /dev/null && debian-sa1 1 1)
> Oct 20 08:13:15 fmslnx0 kernel: [ 2771.978705] nvmet_rdma: connect request
> (4): status 0 id ffff8e0aa2f2bc00
> Oct 20 08:13:15 fmslnx0 kernel: [ 2771.979382] nvmet_rdma: added mlx5_1.
> Oct 20 08:13:15 fmslnx0 kernel: [ 2771.980211]
> mlx5_1:mlx5_ib_create_cq:948:(pid 1442): cqn 0x5d
> Oct 20 08:13:15 fmslnx0 kernel: [ 2771.980269] mlx5_1:calc_sq_size:355:(pid
> 1442): wqe_size 640
> Oct 20 08:13:15 fmslnx0 kernel: [ 2771.981626]
> mlx5_1:mlx5_ib_create_qp:2041:(pid 1442): ib qpnum 0xf1, mlx qpn 0xf1, rcqn
> 0x5d, scqn 0x5d
> Oct 20 08:13:15 fmslnx0 kernel: [ 2771.982372] nvmet_rdma:
> nvmet_rdma_create_queue_ib: max_cqe= 63 max_sge= 32 sq_size = 51 cm_id=
> ffff8e0aa2f2bc00
> Oct 20 08:13:15 fmslnx0 kernel: [ 2771.984271] mlx5_1:poll_soft_wc:647:(pid
> 3422): polled software generated completion on CQ 0x14
> Oct 20 08:13:15 fmslnx0 kernel: [ 2771.985964] nvmet_rdma: established (9):
> status 0 id ffff8e0aa2f2bc00
> Oct 20 08:13:15 fmslnx0 kernel: [ 2771.987073] nvmet: ctrl 1 start
> keep-alive timer for 120 secs
> Oct 20 08:13:15 fmslnx0 kernel: [ 2771.987122] nvmet: creating controller 1
> for NQN nqn.2014-08.org.nvmexpress:NVMf:uuid:01020304.
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.120199] nvmet_rdma: disconnected
> (10): status 0 id ffff8e0aa2f2bc00
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.120253] nvmet_rdma: cm_id=
> ffff8e0aa2f2bc00 queue->state= 1
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.121781] mlx5_1:poll_soft_wc:647:(pid
> 3422): polled software generated completion on CQ 0x14
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122046] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122105] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf5
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122315] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Requestor error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122374] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf5
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122428] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122480] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122534] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122586] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122639] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122690] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122742] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122794] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122846] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122897] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.122949] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123000] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123052] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123103] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123155] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123207] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123260] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123311] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123363] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123414] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123466] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123518] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123570] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123621] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123673] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123724] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123776] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123828] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123881] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123933] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.123991] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.124042] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.124433] mlx5_1:mlx5_poll_one:586:(pid
> 3422): Responder error cqe on cqn 0x5d:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.124488] mlx5_1:mlx5_poll_one:588:(pid
> 3422): syndrome 0x5, vendor syndrome 0xf9
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.124567] nvmet_rdma: freeing queue 0
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.124607] ------------[ cut here
> ]------------
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.124651] WARNING: CPU: 0 PID: 1445 at
> kernel/softirq.c:150 __local_bh_enable_ip+0x6b/0x80
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.124710] Modules linked in: mlx5_ib
> mlx5_core rdma_ucm ib_uverbs ib_mthca nvmet_rdma nvmet nls_iso8859_1
> intel_rapl sb_edac edac_core x86_pkg_temp_thermal intel_powerclamp coretemp
> dcdbas dell_smm_hwmon kvm_intel kvm irqbypass serio_raw
> snd_hda_codec_realtek snd_hda_codec_generic joydev input_leds
> snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep
> snd_pcm lpc_ich snd_timer snd soundcore mei_me shpchp mei ib_iser rdma_cm
> iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
> autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq
> async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear
> hid_generic usbhid hid amdkfd amd_iommu_v2 radeon crct10dif_pclmul
> crc32_pclmul i2c_algo_bit ghash_clmulni_intel ttm aesni_intel drm_kms_helper
> aes_x86_64 lrw gf128mul syscopyarea glue_helper sysfillrect ablk_helper
> sysimgblt cryptd fb_sys_fops psmouse e1000e isci ahci ptp drm libahci nvme
> libsas pps_core nvme_core scsi_transport_sas fjes jitterentropy_rng drbg
> ansi_cprng [last unloaded: mlx5_core]
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.138833] CPU: 0 PID: 1445 Comm:
> kworker/0:29 Not tainted 4.8.2 #1
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.141298] Hardware name: Dell Inc.
> Precision T7600/0VHRW1, BIOS A12 09/29/2014
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.143795] Workqueue: ib_cm
> cm_work_handler [ib_cm]
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.146279] 0000000000000086
> 00000000d356260a ffff8e0ac44478f8 ffffffffb93dfce3
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.148788] 0000000000000000
> 0000000000000000 ffff8e0ac4447938 ffffffffb907899b
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.151285] 00000096d356260a
> 0000000000000200 ffff8e0abb9b0000 ffff8e0cae2a6494
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.153789] Call Trace:
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.156253] [<ffffffffb93dfce3>]
> dump_stack+0x63/0x90
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.158692] [<ffffffffb907899b>]
> __warn+0xcb/0xf0
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.161080] [<ffffffffb9078acd>]
> warn_slowpath_null+0x1d/0x20
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.163444] [<ffffffffb907e08b>]
> __local_bh_enable_ip+0x6b/0x80
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.165818] [<ffffffffb975de6c>]
> ipv4_neigh_lookup+0xac/0x130
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.168184] [<ffffffffc086c512>]
> addr_resolve_neigh+0xb2/0x2b0 [ib_core]
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.170524] [<ffffffffc086c91c>]
> addr_resolve+0x20c/0x280 [ib_core]
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.172851] [<ffffffffb93f5a2a>] ?
> find_next_zero_bit+0x1a/0x20
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.175142] [<ffffffffb93e12f9>] ?
> idr_get_empty_slot+0x199/0x3b0
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.177413] [<ffffffffb91e855c>] ?
> kmem_cache_alloc_trace+0xdc/0x1a0
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.179693] [<ffffffffc086cc3c>]
> rdma_resolve_ip+0x18c/0x2c0 [ib_core]
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.181975] [<ffffffffc086c060>] ?
> rdma_addr_register_client+0x30/0x30 [ib_core]
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.184272] [<ffffffffc086d199>]
> rdma_addr_find_l2_eth_by_grh+0x139/0x240 [ib_core]
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.186577] [<ffffffffb902a77c>] ?
> __switch_to+0x2dc/0x700
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.188885] [<ffffffffc086164d>]
> ib_init_ah_from_wc+0x19d/0x570 [ib_core]
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.191212] [<ffffffffb9034ec9>] ?
> sched_clock+0x9/0x10
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.193531] [<ffffffffb90a8f3f>] ?
> sched_clock_cpu+0x8f/0xa0
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.195846] [<ffffffffb90a2cb4>] ?
> check_preempt_curr+0x54/0x90
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.198162] [<ffffffffb90ac453>] ?
> update_curr+0xf3/0x180
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.200471] [<ffffffffc0861a59>]
> ib_create_ah_from_wc+0x39/0x70 [ib_core]
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.202794] [<ffffffffc0672fc7>]
> cm_alloc_response_msg.isra.33+0x37/0xb0 [ib_cm]
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.205134] [<ffffffffc0677da5>]
> cm_work_handler+0x11d5/0x16f2 [ib_cm]
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.207476] [<ffffffffb9092c4b>]
> process_one_work+0x16b/0x480
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.209815] [<ffffffffb9092fab>]
> worker_thread+0x4b/0x500
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.212157] [<ffffffffb9092f60>] ?
> process_one_work+0x480/0x480
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.214506] [<ffffffffb9099158>]
> kthread+0xd8/0xf0
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.216852] [<ffffffffb9832e1f>]
> ret_from_fork+0x1f/0x40
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.219189] [<ffffffffb9099080>] ?
> kthread_create_on_node+0x1a0/0x1a0
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.221540] ---[ end trace
> 40812fc5b7bae90e ]---
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.223991] mlx5_1:poll_soft_wc:647:(pid
> 3422): polled software generated completion on CQ 0x14
> Oct 20 08:13:15 fmslnx0 kernel: [ 2772.232114] nvmet: ctrl 1 stop keep-alive
>
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
--
Robert Randall | robert.r.randall-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 6/8] IB/hns: Replace counting semaphore event_sem with wait_event
From: Binoy Jayan @ 2016-10-25 13:35 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Doug Ledford, Sean Hefty, Hal Rosenstock, linux-rdma,
Linux kernel mailing list
In-Reply-To: <3400101.E3rphEBHnx@wuerfel>
On 25 October 2016 at 18:51, Arnd Bergmann <arnd@arndb.de> wrote:
> On Tuesday, October 25, 2016 6:29:45 PM CEST Binoy Jayan wrote:
>
> Something like
>
> static struct hns_roce_cmd_context *hns_roce_try_get_context(struct hns_roce_cmdq *cmd)
> {
> struct hns_roce_cmd_context *context = NULL;
>
> spin_lock(&cmd->context_lock);
>
> if (cmd->free_head < 0)
> goto out;
>
> context = &cmd->context[cmd->free_head];
>
> ... /* update free_head */
>
> out:
> spin_unlock(&cmd->context_lock);
>
> return context;
> }
> ...
>
> static struct hns_roce_cmd_context *hns_roce_get_context(struct hns_roce_cmdq *cmd)
> {
> struct hns_roce_cmd_context *context;
>
> wait_event(cmd->wq, (context = hns_roce_try_get_context(cmd)));
>
> return context;
> }
That looks more elegant. Didn't think of that, Thank you Arnd.:)
^ permalink raw reply
* Re: [PATCH v2 6/8] IB/hns: Replace counting semaphore event_sem with wait_event
From: Arnd Bergmann @ 2016-10-25 13:21 UTC (permalink / raw)
To: Binoy Jayan
Cc: Doug Ledford, Sean Hefty, Hal Rosenstock, linux-rdma,
Linux kernel mailing list
In-Reply-To: <CAHv-k__yWEaBhNYqvdjZPFpVtMKGMC75_eMh7Y4V_=MNhtzMbw@mail.gmail.com>
On Tuesday, October 25, 2016 6:29:45 PM CEST Binoy Jayan wrote:
> On 25 October 2016 at 17:58, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Tuesday, October 25, 2016 5:31:57 PM CEST Binoy Jayan wrote:
> >> static int __hns_roce_cmd_mbox_wait(struct hns_roce_dev *hr_dev, u64 in_param,
> >> u64 out_param, unsigned long in_modifier,
> >> @@ -198,11 +218,12 @@ static int __hns_roce_cmd_mbox_wait(struct hns_roce_dev *hr_dev, u64 in_param,
> >> struct hns_roce_cmdq *cmd = &hr_dev->cmd;
> >> struct device *dev = &hr_dev->pdev->dev;
> >> struct hns_roce_cmd_context *context;
> >> - int ret = 0;
> >> + int orig_free_head, ret = 0;
> >> +
> >> + wait_event(cmd->wq, (orig_free_head = atomic_free_node(cmd, -1)) != -1);
> >>
> >> spin_lock(&cmd->context_lock);
> >> - WARN_ON(cmd->free_head < 0);
> >> - context = &cmd->context[cmd->free_head];
> >> + context = &cmd->context[orig_free_head];
> >> context->token += cmd->token_mask + 1;
> >> cmd->free_head = context->next;
> >> spin_unlock(&cmd->context_lock);
> >>
> >
> > You get the lock in atomic_free_node() and then again right after that.
> > Why not combine the two and only take the lock inside of that
> > function that returns a context?
>
>
> Hi Arnd,
>
> I couldn't figure out a way to wait for a node to be free followed by
> acquiring a lock
> in an atomic fashion. If the lock is acquired after the wait_event,
> there could be race
> between the wait_event and acquiring the lock. If the lock is acquired
> before the
> wait_event, the process may goto sleep with the lock held which is not desired.
> Could you suggest me of some way to circumvent this?
Something like
static struct hns_roce_cmd_context *hns_roce_try_get_context(struct hns_roce_cmdq *cmd)
{
struct hns_roce_cmd_context *context = NULL;
spin_lock(&cmd->context_lock);
if (cmd->free_head < 0)
goto out;
context = &cmd->context[cmd->free_head];
... /* update free_head */
out:
spin_unlock(&cmd->context_lock);
return context;
}
...
static struct hns_roce_cmd_context *hns_roce_get_context(struct hns_roce_cmdq *cmd)
{
struct hns_roce_cmd_context *context;
wait_event(cmd->wq, (context = hns_roce_try_get_context(cmd)));
return context;
}
Arnd
^ permalink raw reply
* Re: [PATCH v2 8/8] IB/mlx5: Add helper mlx5_ib_post_send_wait
From: Binoy Jayan @ 2016-10-25 13:16 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Doug Ledford, Sean Hefty, Hal Rosenstock, Arnd Bergmann,
linux-rdma, Linux kernel mailing list
In-Reply-To: <20161025122609.GQ25013@leon.nu>
On 25 October 2016 at 17:56, Leon Romanovsky <leon@kernel.org> wrote:
> On Tue, Oct 25, 2016 at 05:31:59PM +0530, Binoy Jayan wrote:
> In case of success (err == 0), you will call to unmap_dma instead of
> normal flow.
>
> NAK,
> Leon Romanovsky <leonro@mellanox.com>
Hi Loen,
Even in the original code, the regular flow seems to reach 'unmap_dma' after
returning from 'wait_for_completion'().
-Binoy
^ permalink raw reply
* Re: [PATCH v2 6/8] IB/hns: Replace counting semaphore event_sem with wait_event
From: Binoy Jayan @ 2016-10-25 12:59 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Doug Ledford, Sean Hefty, Hal Rosenstock, linux-rdma,
Linux kernel mailing list
In-Reply-To: <5375848.bxLv0aDzDv@wuerfel>
On 25 October 2016 at 17:58, Arnd Bergmann <arnd@arndb.de> wrote:
> On Tuesday, October 25, 2016 5:31:57 PM CEST Binoy Jayan wrote:
>> static int __hns_roce_cmd_mbox_wait(struct hns_roce_dev *hr_dev, u64 in_param,
>> u64 out_param, unsigned long in_modifier,
>> @@ -198,11 +218,12 @@ static int __hns_roce_cmd_mbox_wait(struct hns_roce_dev *hr_dev, u64 in_param,
>> struct hns_roce_cmdq *cmd = &hr_dev->cmd;
>> struct device *dev = &hr_dev->pdev->dev;
>> struct hns_roce_cmd_context *context;
>> - int ret = 0;
>> + int orig_free_head, ret = 0;
>> +
>> + wait_event(cmd->wq, (orig_free_head = atomic_free_node(cmd, -1)) != -1);
>>
>> spin_lock(&cmd->context_lock);
>> - WARN_ON(cmd->free_head < 0);
>> - context = &cmd->context[cmd->free_head];
>> + context = &cmd->context[orig_free_head];
>> context->token += cmd->token_mask + 1;
>> cmd->free_head = context->next;
>> spin_unlock(&cmd->context_lock);
>>
>
> You get the lock in atomic_free_node() and then again right after that.
> Why not combine the two and only take the lock inside of that
> function that returns a context?
Hi Arnd,
I couldn't figure out a way to wait for a node to be free followed by
acquiring a lock
in an atomic fashion. If the lock is acquired after the wait_event,
there could be race
between the wait_event and acquiring the lock. If the lock is acquired
before the
wait_event, the process may goto sleep with the lock held which is not desired.
Could you suggest me of some way to circumvent this?
-Binoy
^ permalink raw reply
* [PATCH rdma-core] qede: fix general protection fault may occur on probe
From: Ram Amrani @ 2016-10-25 12:53 UTC (permalink / raw)
To: dledford-H+wXaHxf7aLQT0dZR+AlfA
Cc: Ariel.Elior-YGCgFSpz5w/QT0dZR+AlfA,
Michal.Kalderon-YGCgFSpz5w/QT0dZR+AlfA,
Yuval.Mintz-YGCgFSpz5w/QT0dZR+AlfA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Ram Amrani
The recent introduction of qedr driver support in qede causes a
GPF when probing the driver in a server without a RoCE enabled
QLogic NIC. This fix avoids using an uninitialized pointer in such
a case. Caught by the kernel test robot.
Signed-off-by: Ram Amrani <Ram.Amrani-YGCgFSpz5w/QT0dZR+AlfA@public.gmane.org>
---
drivers/net/ethernet/qlogic/qede/qede_roce.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/qlogic/qede/qede_roce.c b/drivers/net/ethernet/qlogic/qede/qede_roce.c
index 9867f96..4927271 100644
--- a/drivers/net/ethernet/qlogic/qede/qede_roce.c
+++ b/drivers/net/ethernet/qlogic/qede/qede_roce.c
@@ -191,8 +191,8 @@ int qede_roce_register_driver(struct qedr_driver *drv)
}
mutex_unlock(&qedr_dev_list_lock);
- DP_INFO(edev, "qedr: discovered and registered %d RoCE funcs\n",
- qedr_counter);
+ pr_notice("qedr: discovered and registered %d RoCE funcs\n",
+ qedr_counter);
return 0;
}
--
1.8.3.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH v2 8/8] IB/mlx5: Add helper mlx5_ib_post_send_wait
From: Binoy Jayan @ 2016-10-25 12:46 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Doug Ledford, Sean Hefty, Hal Rosenstock,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux kernel mailing list
In-Reply-To: <5098701.SVmQeRIiho@wuerfel>
On 25 October 2016 at 17:53, Arnd Bergmann <arnd-r2nGTMty4D4@public.gmane.org> wrote:
> On Tuesday, October 25, 2016 5:31:59 PM CEST Binoy Jayan wrote:
> Looks reasonable.
Thank you Arnd for looking at it again.
> Did you get a warning about 'bad' being unused here? I would have
> guessed not, since the original code was not that different and
> it does get passed into a function.
I remembered getting a warning like that, but when i remove the unused
attribute now, it does not. It was probably a side effect of some other error.
Will change it.
> Why not move the umr_context variable into this function too?
> The only thing we ever seem to do to is initialize it and
> assign the wr_cqe pointer, both of which can be done here.
I guess it could be done.
Binoy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 2/8] IB/core: Replace semaphore sm_sem with completion
From: Jack Wang @ 2016-10-25 12:43 UTC (permalink / raw)
To: Binoy Jayan
Cc: Doug Ledford, Sean Hefty, Hal Rosenstock, Arnd Bergmann,
linux-rdma@vger.kernel.org, linux-kernel
In-Reply-To: <1477396919-27669-3-git-send-email-binoy.jayan@linaro.org>
Hi Binoy,
snip
>
> port->ib_dev = device;
> port->port_num = port_num;
> - sema_init(&port->sm_sem, 1);
> + init_completion(&port->sm_comp);
> + complete(&port->sm_comp);
Why complete here?
> mutex_init(&port->file_mutex);
> INIT_LIST_HEAD(&port->file_list);
>
> --
KR,
Jinpu
^ permalink raw reply
* RE: [PATCH rdma-core 4/6] libqedr: main
From: Amrani, Ram @ 2016-10-25 12:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Elior, Ariel,
Kalderon, Michal
In-Reply-To: <20161020170828.GC28181-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
I've just submitted fixes via github.
Thanks for the feedback,
Ram
> -----Original Message-----
> From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
> Sent: Thursday, October 20, 2016 8:08 PM
> To: Amrani, Ram <Ram.Amrani-YGCgFSpz5w/QT0dZR+AlfA@public.gmane.org>
> Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Elior, Ariel
> <Ariel.Elior-YGCgFSpz5w/QT0dZR+AlfA@public.gmane.org>; Kalderon, Michal <Michal.Kalderon-YGCgFSpz5w/QT0dZR+AlfA@public.gmane.org>
> Subject: Re: [PATCH rdma-core 4/6] libqedr: main
>
> On Thu, Oct 20, 2016 at 12:49:10PM +0300, Ram Amrani wrote:
> > +struct {
> > + unsigned int vendor;
> > + unsigned int device;
> > +} hca_table[] = {
>
> needs static const, please check all your stuff for static and const..
>
> > +int qelr_modify_qp(struct ibv_qp *, struct ibv_qp_attr *,
> > + int ibv_qp_attr_mask);
> > +int qelr_query_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr, int attr_mask,
> > + struct ibv_qp_init_attr *init_attr);
>
> It would be nice to be consistent, I prefer the argument name to be in the
> prototype.
>
> Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 6/8] IB/hns: Replace counting semaphore event_sem with wait_event
From: Arnd Bergmann @ 2016-10-25 12:28 UTC (permalink / raw)
To: Binoy Jayan
Cc: Doug Ledford, Sean Hefty, Hal Rosenstock,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1477396919-27669-7-git-send-email-binoy.jayan-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
On Tuesday, October 25, 2016 5:31:57 PM CEST Binoy Jayan wrote:
> static int __hns_roce_cmd_mbox_wait(struct hns_roce_dev *hr_dev, u64 in_param,
> u64 out_param, unsigned long in_modifier,
> @@ -198,11 +218,12 @@ static int __hns_roce_cmd_mbox_wait(struct hns_roce_dev *hr_dev, u64 in_param,
> struct hns_roce_cmdq *cmd = &hr_dev->cmd;
> struct device *dev = &hr_dev->pdev->dev;
> struct hns_roce_cmd_context *context;
> - int ret = 0;
> + int orig_free_head, ret = 0;
> +
> + wait_event(cmd->wq, (orig_free_head = atomic_free_node(cmd, -1)) != -1);
>
> spin_lock(&cmd->context_lock);
> - WARN_ON(cmd->free_head < 0);
> - context = &cmd->context[cmd->free_head];
> + context = &cmd->context[orig_free_head];
> context->token += cmd->token_mask + 1;
> cmd->free_head = context->next;
> spin_unlock(&cmd->context_lock);
>
You get the lock in atomic_free_node() and then again right after that.
Why not combine the two and only take the lock inside of that
function that returns a context?
Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 8/8] IB/mlx5: Add helper mlx5_ib_post_send_wait
From: Leon Romanovsky @ 2016-10-25 12:26 UTC (permalink / raw)
To: Binoy Jayan
Cc: Doug Ledford, Sean Hefty, Hal Rosenstock, Arnd Bergmann,
linux-rdma, linux-kernel
In-Reply-To: <1477396919-27669-9-git-send-email-binoy.jayan@linaro.org>
[-- Attachment #1: Type: text/plain, Size: 7079 bytes --]
On Tue, Oct 25, 2016 at 05:31:59PM +0530, Binoy Jayan wrote:
> Clean up common code (to post a list of work requests to the send queue of
> the specified QP) at various places and add a helper function
> 'mlx5_ib_post_send_wait' to implement the same. The counting semaphore
> 'umr_common:sem' is also moved into the helper. This may later be modified
> to replace the semaphore with an alternative.
>
> Signed-off-by: Binoy Jayan <binoy.jayan@linaro.org>
> ---
> drivers/infiniband/hw/mlx5/mr.c | 96 +++++++++++++----------------------------
> 1 file changed, 29 insertions(+), 67 deletions(-)
>
> diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
> index d4ad672..261984b 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -856,16 +856,38 @@ static inline void mlx5_ib_init_umr_context(struct mlx5_ib_umr_context *context)
> init_completion(&context->done);
> }
>
> +static inline int mlx5_ib_post_send_wait(struct mlx5_ib_dev *dev,
> + struct mlx5_ib_umr_context *umr_context,
> + struct mlx5_umr_wr *umrwr)
> +{
> + struct umr_common *umrc = &dev->umrc;
> + struct ib_send_wr __maybe_unused *bad;
> + int err;
> +
> + down(&umrc->sem);
> + err = ib_post_send(umrc->qp, &umrwr->wr, &bad);
> + if (err) {
> + mlx5_ib_warn(dev, "UMR post send failed, err %d\n", err);
> + } else {
> + wait_for_completion(&umr_context->done);
> + if (umr_context->status != IB_WC_SUCCESS) {
> + mlx5_ib_warn(dev, "reg umr failed (%u)\n",
> + umr_context->status);
> + err = -EFAULT;
> + }
> + }
> + up(&umrc->sem);
> + return err;
> +}
> +
> static struct mlx5_ib_mr *reg_umr(struct ib_pd *pd, struct ib_umem *umem,
> u64 virt_addr, u64 len, int npages,
> int page_shift, int order, int access_flags)
> {
> struct mlx5_ib_dev *dev = to_mdev(pd->device);
> struct device *ddev = dev->ib_dev.dma_device;
> - struct umr_common *umrc = &dev->umrc;
> struct mlx5_ib_umr_context umr_context;
> struct mlx5_umr_wr umrwr = {};
> - struct ib_send_wr *bad;
> struct mlx5_ib_mr *mr;
> struct ib_sge sg;
> int size;
> @@ -900,18 +922,9 @@ static struct mlx5_ib_mr *reg_umr(struct ib_pd *pd, struct ib_umem *umem,
> prep_umr_reg_wqe(pd, &umrwr.wr, &sg, dma, npages, mr->mmkey.key,
> page_shift, virt_addr, len, access_flags);
>
> - down(&umrc->sem);
> - err = ib_post_send(umrc->qp, &umrwr.wr, &bad);
> - if (err) {
> - mlx5_ib_warn(dev, "post send failed, err %d\n", err);
> + err = mlx5_ib_post_send_wait(dev, &umr_context, &umrwr);
> + if (err != -EFAULT)
> goto unmap_dma;
In case of success (err == 0), you will call to unmap_dma instead of
normal flow.
NAK,
Leon Romanovsky <leonro@mellanox.com>
> - } else {
> - wait_for_completion(&umr_context.done);
> - if (umr_context.status != IB_WC_SUCCESS) {
> - mlx5_ib_warn(dev, "reg umr failed\n");
> - err = -EFAULT;
> - }
> - }
>
> mr->mmkey.iova = virt_addr;
> mr->mmkey.size = len;
> @@ -920,7 +933,6 @@ static struct mlx5_ib_mr *reg_umr(struct ib_pd *pd, struct ib_umem *umem,
> mr->live = 1;
>
> unmap_dma:
> - up(&umrc->sem);
> dma_unmap_single(ddev, dma, size, DMA_TO_DEVICE);
>
> kfree(mr_pas);
> @@ -940,13 +952,11 @@ int mlx5_ib_update_mtt(struct mlx5_ib_mr *mr, u64 start_page_index, int npages,
> {
> struct mlx5_ib_dev *dev = mr->dev;
> struct device *ddev = dev->ib_dev.dma_device;
> - struct umr_common *umrc = &dev->umrc;
> struct mlx5_ib_umr_context umr_context;
> struct ib_umem *umem = mr->umem;
> int size;
> __be64 *pas;
> dma_addr_t dma;
> - struct ib_send_wr *bad;
> struct mlx5_umr_wr wr;
> struct ib_sge sg;
> int err = 0;
> @@ -1031,19 +1041,7 @@ int mlx5_ib_update_mtt(struct mlx5_ib_mr *mr, u64 start_page_index, int npages,
> wr.mkey = mr->mmkey.key;
> wr.target.offset = start_page_index;
>
> - down(&umrc->sem);
> - err = ib_post_send(umrc->qp, &wr.wr, &bad);
> - if (err) {
> - mlx5_ib_err(dev, "UMR post send failed, err %d\n", err);
> - } else {
> - wait_for_completion(&umr_context.done);
> - if (umr_context.status != IB_WC_SUCCESS) {
> - mlx5_ib_err(dev, "UMR completion failed, code %d\n",
> - umr_context.status);
> - err = -EFAULT;
> - }
> - }
> - up(&umrc->sem);
> + err = mlx5_ib_post_send_wait(dev, &umr_context, &wr);
> }
> dma_unmap_single(ddev, dma, size, DMA_TO_DEVICE);
>
> @@ -1210,11 +1208,8 @@ struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
> static int unreg_umr(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr)
> {
> struct mlx5_core_dev *mdev = dev->mdev;
> - struct umr_common *umrc = &dev->umrc;
> struct mlx5_ib_umr_context umr_context;
> struct mlx5_umr_wr umrwr = {};
> - struct ib_send_wr *bad;
> - int err;
>
> if (mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR)
> return 0;
> @@ -1224,25 +1219,7 @@ static int unreg_umr(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr)
> umrwr.wr.wr_cqe = &umr_context.cqe;
> prep_umr_unreg_wqe(dev, &umrwr.wr, mr->mmkey.key);
>
> - down(&umrc->sem);
> - err = ib_post_send(umrc->qp, &umrwr.wr, &bad);
> - if (err) {
> - up(&umrc->sem);
> - mlx5_ib_dbg(dev, "err %d\n", err);
> - goto error;
> - } else {
> - wait_for_completion(&umr_context.done);
> - up(&umrc->sem);
> - }
> - if (umr_context.status != IB_WC_SUCCESS) {
> - mlx5_ib_warn(dev, "unreg umr failed\n");
> - err = -EFAULT;
> - goto error;
> - }
> - return 0;
> -
> -error:
> - return err;
> + return mlx5_ib_post_send_wait(dev, &umr_context, &umrwr);
> }
>
> static int rereg_umr(struct ib_pd *pd, struct mlx5_ib_mr *mr, u64 virt_addr,
> @@ -1252,10 +1229,8 @@ static int rereg_umr(struct ib_pd *pd, struct mlx5_ib_mr *mr, u64 virt_addr,
> struct mlx5_ib_dev *dev = to_mdev(pd->device);
> struct device *ddev = dev->ib_dev.dma_device;
> struct mlx5_ib_umr_context umr_context;
> - struct ib_send_wr *bad;
> struct mlx5_umr_wr umrwr = {};
> struct ib_sge sg;
> - struct umr_common *umrc = &dev->umrc;
> dma_addr_t dma = 0;
> __be64 *mr_pas = NULL;
> int size;
> @@ -1291,21 +1266,8 @@ static int rereg_umr(struct ib_pd *pd, struct mlx5_ib_mr *mr, u64 virt_addr,
> }
>
> /* post send request to UMR QP */
> - down(&umrc->sem);
> - err = ib_post_send(umrc->qp, &umrwr.wr, &bad);
> -
> - if (err) {
> - mlx5_ib_warn(dev, "post send failed, err %d\n", err);
> - } else {
> - wait_for_completion(&umr_context.done);
> - if (umr_context.status != IB_WC_SUCCESS) {
> - mlx5_ib_warn(dev, "reg umr failed (%u)\n",
> - umr_context.status);
> - err = -EFAULT;
> - }
> - }
> + err = mlx5_ib_post_send_wait(dev, &umr_context, &umrwr);
>
> - up(&umrc->sem);
> if (flags & IB_MR_REREG_TRANS) {
> dma_unmap_single(ddev, dma, size, DMA_TO_DEVICE);
> kfree(mr_pas);
> --
> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> a Linux Foundation Collaborative Project
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply
* Re: [PATCH v2 8/8] IB/mlx5: Add helper mlx5_ib_post_send_wait
From: Arnd Bergmann @ 2016-10-25 12:23 UTC (permalink / raw)
To: Binoy Jayan
Cc: Doug Ledford, Sean Hefty, Hal Rosenstock, linux-rdma,
linux-kernel
In-Reply-To: <1477396919-27669-9-git-send-email-binoy.jayan@linaro.org>
On Tuesday, October 25, 2016 5:31:59 PM CEST Binoy Jayan wrote:
> Clean up common code (to post a list of work requests to the send queue of
> the specified QP) at various places and add a helper function
> 'mlx5_ib_post_send_wait' to implement the same. The counting semaphore
> 'umr_common:sem' is also moved into the helper. This may later be modified
> to replace the semaphore with an alternative.
>
> Signed-off-by: Binoy Jayan <binoy.jayan@linaro.org>
Looks reasonable.
> ---
> drivers/infiniband/hw/mlx5/mr.c | 96 +++++++++++++----------------------------
> 1 file changed, 29 insertions(+), 67 deletions(-)
>
> diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
> index d4ad672..261984b 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -856,16 +856,38 @@ static inline void mlx5_ib_init_umr_context(struct mlx5_ib_umr_context *context)
> init_completion(&context->done);
> }
>
> +static inline int mlx5_ib_post_send_wait(struct mlx5_ib_dev *dev,
> + struct mlx5_ib_umr_context *umr_context,
> + struct mlx5_umr_wr *umrwr)
> +{
> + struct umr_common *umrc = &dev->umrc;
> + struct ib_send_wr __maybe_unused *bad;
Did you get a warning about 'bad' being unused here? I would have
guessed not, since the original code was not that different and
it does get passed into a function.
Why not move the umr_context variable into this function too?
The only thing we ever seem to do to is initialize it and
assign the wr_cqe pointer, both of which can be done here.
Arnd
^ permalink raw reply
* [PATCH v2 8/8] IB/mlx5: Add helper mlx5_ib_post_send_wait
From: Binoy Jayan @ 2016-10-25 12:01 UTC (permalink / raw)
To: Doug Ledford, Sean Hefty, Hal Rosenstock
Cc: Arnd Bergmann, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA, Binoy Jayan
In-Reply-To: <1477396919-27669-1-git-send-email-binoy.jayan-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
Clean up common code (to post a list of work requests to the send queue of
the specified QP) at various places and add a helper function
'mlx5_ib_post_send_wait' to implement the same. The counting semaphore
'umr_common:sem' is also moved into the helper. This may later be modified
to replace the semaphore with an alternative.
Signed-off-by: Binoy Jayan <binoy.jayan-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
---
drivers/infiniband/hw/mlx5/mr.c | 96 +++++++++++++----------------------------
1 file changed, 29 insertions(+), 67 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index d4ad672..261984b 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -856,16 +856,38 @@ static inline void mlx5_ib_init_umr_context(struct mlx5_ib_umr_context *context)
init_completion(&context->done);
}
+static inline int mlx5_ib_post_send_wait(struct mlx5_ib_dev *dev,
+ struct mlx5_ib_umr_context *umr_context,
+ struct mlx5_umr_wr *umrwr)
+{
+ struct umr_common *umrc = &dev->umrc;
+ struct ib_send_wr __maybe_unused *bad;
+ int err;
+
+ down(&umrc->sem);
+ err = ib_post_send(umrc->qp, &umrwr->wr, &bad);
+ if (err) {
+ mlx5_ib_warn(dev, "UMR post send failed, err %d\n", err);
+ } else {
+ wait_for_completion(&umr_context->done);
+ if (umr_context->status != IB_WC_SUCCESS) {
+ mlx5_ib_warn(dev, "reg umr failed (%u)\n",
+ umr_context->status);
+ err = -EFAULT;
+ }
+ }
+ up(&umrc->sem);
+ return err;
+}
+
static struct mlx5_ib_mr *reg_umr(struct ib_pd *pd, struct ib_umem *umem,
u64 virt_addr, u64 len, int npages,
int page_shift, int order, int access_flags)
{
struct mlx5_ib_dev *dev = to_mdev(pd->device);
struct device *ddev = dev->ib_dev.dma_device;
- struct umr_common *umrc = &dev->umrc;
struct mlx5_ib_umr_context umr_context;
struct mlx5_umr_wr umrwr = {};
- struct ib_send_wr *bad;
struct mlx5_ib_mr *mr;
struct ib_sge sg;
int size;
@@ -900,18 +922,9 @@ static struct mlx5_ib_mr *reg_umr(struct ib_pd *pd, struct ib_umem *umem,
prep_umr_reg_wqe(pd, &umrwr.wr, &sg, dma, npages, mr->mmkey.key,
page_shift, virt_addr, len, access_flags);
- down(&umrc->sem);
- err = ib_post_send(umrc->qp, &umrwr.wr, &bad);
- if (err) {
- mlx5_ib_warn(dev, "post send failed, err %d\n", err);
+ err = mlx5_ib_post_send_wait(dev, &umr_context, &umrwr);
+ if (err != -EFAULT)
goto unmap_dma;
- } else {
- wait_for_completion(&umr_context.done);
- if (umr_context.status != IB_WC_SUCCESS) {
- mlx5_ib_warn(dev, "reg umr failed\n");
- err = -EFAULT;
- }
- }
mr->mmkey.iova = virt_addr;
mr->mmkey.size = len;
@@ -920,7 +933,6 @@ static struct mlx5_ib_mr *reg_umr(struct ib_pd *pd, struct ib_umem *umem,
mr->live = 1;
unmap_dma:
- up(&umrc->sem);
dma_unmap_single(ddev, dma, size, DMA_TO_DEVICE);
kfree(mr_pas);
@@ -940,13 +952,11 @@ int mlx5_ib_update_mtt(struct mlx5_ib_mr *mr, u64 start_page_index, int npages,
{
struct mlx5_ib_dev *dev = mr->dev;
struct device *ddev = dev->ib_dev.dma_device;
- struct umr_common *umrc = &dev->umrc;
struct mlx5_ib_umr_context umr_context;
struct ib_umem *umem = mr->umem;
int size;
__be64 *pas;
dma_addr_t dma;
- struct ib_send_wr *bad;
struct mlx5_umr_wr wr;
struct ib_sge sg;
int err = 0;
@@ -1031,19 +1041,7 @@ int mlx5_ib_update_mtt(struct mlx5_ib_mr *mr, u64 start_page_index, int npages,
wr.mkey = mr->mmkey.key;
wr.target.offset = start_page_index;
- down(&umrc->sem);
- err = ib_post_send(umrc->qp, &wr.wr, &bad);
- if (err) {
- mlx5_ib_err(dev, "UMR post send failed, err %d\n", err);
- } else {
- wait_for_completion(&umr_context.done);
- if (umr_context.status != IB_WC_SUCCESS) {
- mlx5_ib_err(dev, "UMR completion failed, code %d\n",
- umr_context.status);
- err = -EFAULT;
- }
- }
- up(&umrc->sem);
+ err = mlx5_ib_post_send_wait(dev, &umr_context, &wr);
}
dma_unmap_single(ddev, dma, size, DMA_TO_DEVICE);
@@ -1210,11 +1208,8 @@ struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
static int unreg_umr(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr)
{
struct mlx5_core_dev *mdev = dev->mdev;
- struct umr_common *umrc = &dev->umrc;
struct mlx5_ib_umr_context umr_context;
struct mlx5_umr_wr umrwr = {};
- struct ib_send_wr *bad;
- int err;
if (mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR)
return 0;
@@ -1224,25 +1219,7 @@ static int unreg_umr(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr)
umrwr.wr.wr_cqe = &umr_context.cqe;
prep_umr_unreg_wqe(dev, &umrwr.wr, mr->mmkey.key);
- down(&umrc->sem);
- err = ib_post_send(umrc->qp, &umrwr.wr, &bad);
- if (err) {
- up(&umrc->sem);
- mlx5_ib_dbg(dev, "err %d\n", err);
- goto error;
- } else {
- wait_for_completion(&umr_context.done);
- up(&umrc->sem);
- }
- if (umr_context.status != IB_WC_SUCCESS) {
- mlx5_ib_warn(dev, "unreg umr failed\n");
- err = -EFAULT;
- goto error;
- }
- return 0;
-
-error:
- return err;
+ return mlx5_ib_post_send_wait(dev, &umr_context, &umrwr);
}
static int rereg_umr(struct ib_pd *pd, struct mlx5_ib_mr *mr, u64 virt_addr,
@@ -1252,10 +1229,8 @@ static int rereg_umr(struct ib_pd *pd, struct mlx5_ib_mr *mr, u64 virt_addr,
struct mlx5_ib_dev *dev = to_mdev(pd->device);
struct device *ddev = dev->ib_dev.dma_device;
struct mlx5_ib_umr_context umr_context;
- struct ib_send_wr *bad;
struct mlx5_umr_wr umrwr = {};
struct ib_sge sg;
- struct umr_common *umrc = &dev->umrc;
dma_addr_t dma = 0;
__be64 *mr_pas = NULL;
int size;
@@ -1291,21 +1266,8 @@ static int rereg_umr(struct ib_pd *pd, struct mlx5_ib_mr *mr, u64 virt_addr,
}
/* post send request to UMR QP */
- down(&umrc->sem);
- err = ib_post_send(umrc->qp, &umrwr.wr, &bad);
-
- if (err) {
- mlx5_ib_warn(dev, "post send failed, err %d\n", err);
- } else {
- wait_for_completion(&umr_context.done);
- if (umr_context.status != IB_WC_SUCCESS) {
- mlx5_ib_warn(dev, "reg umr failed (%u)\n",
- umr_context.status);
- err = -EFAULT;
- }
- }
+ err = mlx5_ib_post_send_wait(dev, &umr_context, &umrwr);
- up(&umrc->sem);
if (flags & IB_MR_REREG_TRANS) {
dma_unmap_single(ddev, dma, size, DMA_TO_DEVICE);
kfree(mr_pas);
--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH v2 7/8] IB/mthca: Replace counting semaphore event_sem with wait_event
From: Binoy Jayan @ 2016-10-25 12:01 UTC (permalink / raw)
To: Doug Ledford, Sean Hefty, Hal Rosenstock
Cc: Arnd Bergmann, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA, Binoy Jayan
In-Reply-To: <1477396919-27669-1-git-send-email-binoy.jayan-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
Counting semaphores are going away in the future, so replace the semaphore
mthca_cmd::event_sem with a conditional wait_event.
Signed-off-by: Binoy Jayan <binoy.jayan-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
---
drivers/infiniband/hw/mthca/mthca_cmd.c | 37 ++++++++++++++++++++++++---------
drivers/infiniband/hw/mthca/mthca_dev.h | 3 ++-
2 files changed, 29 insertions(+), 11 deletions(-)
diff --git a/drivers/infiniband/hw/mthca/mthca_cmd.c b/drivers/infiniband/hw/mthca/mthca_cmd.c
index 49c6e19..87c475c 100644
--- a/drivers/infiniband/hw/mthca/mthca_cmd.c
+++ b/drivers/infiniband/hw/mthca/mthca_cmd.c
@@ -405,6 +405,26 @@ void mthca_cmd_event(struct mthca_dev *dev,
complete(&context->done);
}
+/* Similar to atomic_cmpxchg but with the complimentary condition. Returns
+ * index to a free node. It also sets cmd->free_head to 'new' so it ensures
+ * atomicity between a call to 'wait_event' and manipulating the free_head.
+ */
+
+static inline int atomic_free_node(struct mthca_cmd *cmd, int new)
+{
+ int orig;
+
+ spin_lock(&cmd->context_lock);
+
+ orig = cmd->free_head;
+ if (likely(cmd->free_head != -1))
+ cmd->free_head = new;
+
+ spin_unlock(&cmd->context_lock);
+
+ return orig;
+}
+
static int mthca_cmd_wait(struct mthca_dev *dev,
u64 in_param,
u64 *out_param,
@@ -414,14 +434,14 @@ static int mthca_cmd_wait(struct mthca_dev *dev,
u16 op,
unsigned long timeout)
{
- int err = 0;
+ int orig_free_head, err = 0;
struct mthca_cmd_context *context;
- down(&dev->cmd.event_sem);
+ wait_event(dev->cmd.wq,
+ (orig_free_head = atomic_free_node(&dev->cmd, -1)) != -1);
spin_lock(&dev->cmd.context_lock);
- BUG_ON(dev->cmd.free_head < 0);
- context = &dev->cmd.context[dev->cmd.free_head];
+ context = &dev->cmd.context[orig_free_head];
context->token += dev->cmd.token_mask + 1;
dev->cmd.free_head = context->next;
spin_unlock(&dev->cmd.context_lock);
@@ -458,8 +478,8 @@ static int mthca_cmd_wait(struct mthca_dev *dev,
context->next = dev->cmd.free_head;
dev->cmd.free_head = context - dev->cmd.context;
spin_unlock(&dev->cmd.context_lock);
+ wake_up(&dev->cmd.wq);
- up(&dev->cmd.event_sem);
return err;
}
@@ -571,7 +591,7 @@ int mthca_cmd_use_events(struct mthca_dev *dev)
dev->cmd.context[dev->cmd.max_cmds - 1].next = -1;
dev->cmd.free_head = 0;
- sema_init(&dev->cmd.event_sem, dev->cmd.max_cmds);
+ init_waitqueue_head(&dev->cmd.wq);
spin_lock_init(&dev->cmd.context_lock);
for (dev->cmd.token_mask = 1;
@@ -590,12 +610,9 @@ int mthca_cmd_use_events(struct mthca_dev *dev)
*/
void mthca_cmd_use_polling(struct mthca_dev *dev)
{
- int i;
-
dev->cmd.flags &= ~MTHCA_CMD_USE_EVENTS;
- for (i = 0; i < dev->cmd.max_cmds; ++i)
- down(&dev->cmd.event_sem);
+ dev->cmd.free_head = -1;
kfree(dev->cmd.context);
}
diff --git a/drivers/infiniband/hw/mthca/mthca_dev.h b/drivers/infiniband/hw/mthca/mthca_dev.h
index 87ab964..2fc86db 100644
--- a/drivers/infiniband/hw/mthca/mthca_dev.h
+++ b/drivers/infiniband/hw/mthca/mthca_dev.h
@@ -46,6 +46,7 @@
#include <linux/list.h>
#include <linux/semaphore.h>
+#include <rdma/ib_sa.h>
#include "mthca_provider.h"
#include "mthca_doorbell.h"
@@ -121,7 +122,7 @@ struct mthca_cmd {
struct pci_pool *pool;
struct mutex hcr_mutex;
struct mutex poll_mutex;
- struct semaphore event_sem;
+ wait_queue_head_t wq;
int max_cmds;
spinlock_t context_lock;
int free_head;
--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH v2 6/8] IB/hns: Replace counting semaphore event_sem with wait_event
From: Binoy Jayan @ 2016-10-25 12:01 UTC (permalink / raw)
To: Doug Ledford, Sean Hefty, Hal Rosenstock
Cc: Arnd Bergmann, linux-rdma, linux-kernel, Binoy Jayan
In-Reply-To: <1477396919-27669-1-git-send-email-binoy.jayan@linaro.org>
Counting semaphores are going away in the future, so replace the semaphore
mthca_cmd::event_sem with a conditional wait_event.
Signed-off-by: Binoy Jayan <binoy.jayan@linaro.org>
---
drivers/infiniband/hw/hns/hns_roce_cmd.c | 37 +++++++++++++++++++++--------
drivers/infiniband/hw/hns/hns_roce_device.h | 2 +-
2 files changed, 28 insertions(+), 11 deletions(-)
diff --git a/drivers/infiniband/hw/hns/hns_roce_cmd.c b/drivers/infiniband/hw/hns/hns_roce_cmd.c
index 51a0675..efc5c48 100644
--- a/drivers/infiniband/hw/hns/hns_roce_cmd.c
+++ b/drivers/infiniband/hw/hns/hns_roce_cmd.c
@@ -189,6 +189,26 @@ void hns_roce_cmd_event(struct hns_roce_dev *hr_dev, u16 token, u8 status,
complete(&context->done);
}
+/* Similar to atomic_cmpxchg but with the complimentary condition. Returns
+ * index to a free node. It also sets cmd->free_head to 'new' so it ensures
+ * atomicity between a call to 'wait_event' and manipulating the free_head.
+ */
+
+static inline int atomic_free_node(struct hns_roce_cmdq *cmd, int new)
+{
+ int orig;
+
+ spin_lock(&cmd->context_lock);
+
+ orig = cmd->free_head;
+ if (likely(cmd->free_head != -1))
+ cmd->free_head = new;
+
+ spin_unlock(&cmd->context_lock);
+
+ return orig;
+}
+
/* this should be called with "use_events" */
static int __hns_roce_cmd_mbox_wait(struct hns_roce_dev *hr_dev, u64 in_param,
u64 out_param, unsigned long in_modifier,
@@ -198,11 +218,12 @@ static int __hns_roce_cmd_mbox_wait(struct hns_roce_dev *hr_dev, u64 in_param,
struct hns_roce_cmdq *cmd = &hr_dev->cmd;
struct device *dev = &hr_dev->pdev->dev;
struct hns_roce_cmd_context *context;
- int ret = 0;
+ int orig_free_head, ret = 0;
+
+ wait_event(cmd->wq, (orig_free_head = atomic_free_node(cmd, -1)) != -1);
spin_lock(&cmd->context_lock);
- WARN_ON(cmd->free_head < 0);
- context = &cmd->context[cmd->free_head];
+ context = &cmd->context[orig_free_head];
context->token += cmd->token_mask + 1;
cmd->free_head = context->next;
spin_unlock(&cmd->context_lock);
@@ -238,6 +259,7 @@ static int __hns_roce_cmd_mbox_wait(struct hns_roce_dev *hr_dev, u64 in_param,
context->next = cmd->free_head;
cmd->free_head = context - cmd->context;
spin_unlock(&cmd->context_lock);
+ wake_up(&cmd->wq);
return ret;
}
@@ -248,10 +270,8 @@ static int hns_roce_cmd_mbox_wait(struct hns_roce_dev *hr_dev, u64 in_param,
{
int ret = 0;
- down(&hr_dev->cmd.event_sem);
ret = __hns_roce_cmd_mbox_wait(hr_dev, in_param, out_param,
in_modifier, op_modifier, op, timeout);
- up(&hr_dev->cmd.event_sem);
return ret;
}
@@ -313,7 +333,7 @@ int hns_roce_cmd_use_events(struct hns_roce_dev *hr_dev)
hr_cmd->context[hr_cmd->max_cmds - 1].next = -1;
hr_cmd->free_head = 0;
- sema_init(&hr_cmd->event_sem, hr_cmd->max_cmds);
+ init_waitqueue_head(&hr_cmd->wq);
spin_lock_init(&hr_cmd->context_lock);
hr_cmd->token_mask = CMD_TOKEN_MASK;
@@ -325,12 +345,9 @@ int hns_roce_cmd_use_events(struct hns_roce_dev *hr_dev)
void hns_roce_cmd_use_polling(struct hns_roce_dev *hr_dev)
{
struct hns_roce_cmdq *hr_cmd = &hr_dev->cmd;
- int i;
hr_cmd->use_events = 0;
-
- for (i = 0; i < hr_cmd->max_cmds; ++i)
- down(&hr_cmd->event_sem);
+ hr_cmd->free_head = -1;
kfree(hr_cmd->context);
}
diff --git a/drivers/infiniband/hw/hns/hns_roce_device.h b/drivers/infiniband/hw/hns/hns_roce_device.h
index 2afe075..ac95f52 100644
--- a/drivers/infiniband/hw/hns/hns_roce_device.h
+++ b/drivers/infiniband/hw/hns/hns_roce_device.h
@@ -364,7 +364,7 @@ struct hns_roce_cmdq {
* Event mode: cmd register mutex protection,
* ensure to not exceed max_cmds and user use limit region
*/
- struct semaphore event_sem;
+ wait_queue_head_t wq;
int max_cmds;
spinlock_t context_lock;
int free_head;
--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
^ permalink raw reply related
* [PATCH v2 5/8] IB/isert: Replace semaphore sem with completion
From: Binoy Jayan @ 2016-10-25 12:01 UTC (permalink / raw)
To: Doug Ledford, Sean Hefty, Hal Rosenstock
Cc: Arnd Bergmann, linux-rdma, linux-kernel, Binoy Jayan
In-Reply-To: <1477396919-27669-1-git-send-email-binoy.jayan@linaro.org>
The semaphore 'sem' in isert_device is used as completion, so convert
it to struct completion. Semaphores are going away in the future.
Signed-off-by: Binoy Jayan <binoy.jayan@linaro.org>
---
drivers/infiniband/ulp/isert/ib_isert.c | 6 +++---
drivers/infiniband/ulp/isert/ib_isert.h | 3 ++-
2 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 6dd43f6..de80f56 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -619,7 +619,7 @@
mutex_unlock(&isert_np->mutex);
isert_info("np %p: Allow accept_np to continue\n", isert_np);
- up(&isert_np->sem);
+ complete(&isert_np->comp);
}
static void
@@ -2311,7 +2311,7 @@ struct rdma_cm_id *
isert_err("Unable to allocate struct isert_np\n");
return -ENOMEM;
}
- sema_init(&isert_np->sem, 0);
+ init_completion(&isert_np->comp);
mutex_init(&isert_np->mutex);
INIT_LIST_HEAD(&isert_np->accepted);
INIT_LIST_HEAD(&isert_np->pending);
@@ -2427,7 +2427,7 @@ struct rdma_cm_id *
int ret;
accept_wait:
- ret = down_interruptible(&isert_np->sem);
+ ret = wait_for_completion_interruptible(&isert_np->comp);
if (ret)
return -ENODEV;
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index c02ada5..a1277c0 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -3,6 +3,7 @@
#include <linux/in6.h>
#include <rdma/ib_verbs.h>
#include <rdma/rdma_cm.h>
+#include <linux/completion.h>
#include <rdma/rw.h>
#include <scsi/iser.h>
@@ -190,7 +191,7 @@ struct isert_device {
struct isert_np {
struct iscsi_np *np;
- struct semaphore sem;
+ struct completion comp;
struct rdma_cm_id *cm_id;
struct mutex mutex;
struct list_head accepted;
--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
^ permalink raw reply related
* [PATCH v2 4/8] IB/mthca: Replace semaphore poll_sem with mutex
From: Binoy Jayan @ 2016-10-25 12:01 UTC (permalink / raw)
To: Doug Ledford, Sean Hefty, Hal Rosenstock
Cc: Arnd Bergmann, linux-rdma, linux-kernel, Binoy Jayan
In-Reply-To: <1477396919-27669-1-git-send-email-binoy.jayan@linaro.org>
The semaphore 'poll_sem' is a simple mutex, so it should be written as one.
Semaphores are going away in the future. So replace the semaphore 'poll_sem'
with a mutex. Also, remove mutex_[un]lock from mthca_cmd_use_events and
mthca_cmd_use_polling respectively.
Signed-off-by: Binoy Jayan <binoy.jayan@linaro.org>
---
drivers/infiniband/hw/mthca/mthca_cmd.c | 10 +++-------
drivers/infiniband/hw/mthca/mthca_cmd.h | 1 +
drivers/infiniband/hw/mthca/mthca_dev.h | 2 +-
3 files changed, 5 insertions(+), 8 deletions(-)
diff --git a/drivers/infiniband/hw/mthca/mthca_cmd.c b/drivers/infiniband/hw/mthca/mthca_cmd.c
index c7f49bb..49c6e19 100644
--- a/drivers/infiniband/hw/mthca/mthca_cmd.c
+++ b/drivers/infiniband/hw/mthca/mthca_cmd.c
@@ -347,7 +347,7 @@ static int mthca_cmd_poll(struct mthca_dev *dev,
unsigned long end;
u8 status;
- down(&dev->cmd.poll_sem);
+ mutex_lock(&dev->cmd.poll_mutex);
err = mthca_cmd_post(dev, in_param,
out_param ? *out_param : 0,
@@ -382,7 +382,7 @@ static int mthca_cmd_poll(struct mthca_dev *dev,
}
out:
- up(&dev->cmd.poll_sem);
+ mutex_unlock(&dev->cmd.poll_mutex);
return err;
}
@@ -520,7 +520,7 @@ static int mthca_cmd_imm(struct mthca_dev *dev,
int mthca_cmd_init(struct mthca_dev *dev)
{
mutex_init(&dev->cmd.hcr_mutex);
- sema_init(&dev->cmd.poll_sem, 1);
+ mutex_init(&dev->cmd.poll_mutex);
dev->cmd.flags = 0;
dev->hcr = ioremap(pci_resource_start(dev->pdev, 0) + MTHCA_HCR_BASE,
@@ -582,8 +582,6 @@ int mthca_cmd_use_events(struct mthca_dev *dev)
dev->cmd.flags |= MTHCA_CMD_USE_EVENTS;
- down(&dev->cmd.poll_sem);
-
return 0;
}
@@ -600,8 +598,6 @@ void mthca_cmd_use_polling(struct mthca_dev *dev)
down(&dev->cmd.event_sem);
kfree(dev->cmd.context);
-
- up(&dev->cmd.poll_sem);
}
struct mthca_mailbox *mthca_alloc_mailbox(struct mthca_dev *dev,
diff --git a/drivers/infiniband/hw/mthca/mthca_cmd.h b/drivers/infiniband/hw/mthca/mthca_cmd.h
index d2e5b19..a7f197e 100644
--- a/drivers/infiniband/hw/mthca/mthca_cmd.h
+++ b/drivers/infiniband/hw/mthca/mthca_cmd.h
@@ -35,6 +35,7 @@
#ifndef MTHCA_CMD_H
#define MTHCA_CMD_H
+#include <linux/mutex.h>
#include <rdma/ib_verbs.h>
#define MTHCA_MAILBOX_SIZE 4096
diff --git a/drivers/infiniband/hw/mthca/mthca_dev.h b/drivers/infiniband/hw/mthca/mthca_dev.h
index 4393a02..87ab964 100644
--- a/drivers/infiniband/hw/mthca/mthca_dev.h
+++ b/drivers/infiniband/hw/mthca/mthca_dev.h
@@ -120,7 +120,7 @@ enum {
struct mthca_cmd {
struct pci_pool *pool;
struct mutex hcr_mutex;
- struct semaphore poll_sem;
+ struct mutex poll_mutex;
struct semaphore event_sem;
int max_cmds;
spinlock_t context_lock;
--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
^ permalink raw reply related
* [PATCH v2 3/8] IB/hns: Replace semaphore poll_sem with mutex
From: Binoy Jayan @ 2016-10-25 12:01 UTC (permalink / raw)
To: Doug Ledford, Sean Hefty, Hal Rosenstock
Cc: Arnd Bergmann, linux-rdma, linux-kernel, Binoy Jayan
In-Reply-To: <1477396919-27669-1-git-send-email-binoy.jayan@linaro.org>
The semaphore 'poll_sem' is a simple mutex, so it should be written as one.
Semaphores are going away in the future. So replace the semaphore 'poll_sem'
with a mutex. Also, remove mutex_[un]lock from mthca_cmd_use_events and
mthca_cmd_use_polling respectively.
Signed-off-by: Binoy Jayan <binoy.jayan@linaro.org>
---
drivers/infiniband/hw/hns/hns_roce_cmd.c | 11 ++++-------
drivers/infiniband/hw/hns/hns_roce_device.h | 3 ++-
2 files changed, 6 insertions(+), 8 deletions(-)
diff --git a/drivers/infiniband/hw/hns/hns_roce_cmd.c b/drivers/infiniband/hw/hns/hns_roce_cmd.c
index 2a0b6c0..51a0675 100644
--- a/drivers/infiniband/hw/hns/hns_roce_cmd.c
+++ b/drivers/infiniband/hw/hns/hns_roce_cmd.c
@@ -119,7 +119,7 @@ static int hns_roce_cmd_mbox_post_hw(struct hns_roce_dev *hr_dev, u64 in_param,
return ret;
}
-/* this should be called with "poll_sem" */
+/* this should be called with "poll_mutex" */
static int __hns_roce_cmd_mbox_poll(struct hns_roce_dev *hr_dev, u64 in_param,
u64 out_param, unsigned long in_modifier,
u8 op_modifier, u16 op,
@@ -167,10 +167,10 @@ static int hns_roce_cmd_mbox_poll(struct hns_roce_dev *hr_dev, u64 in_param,
{
int ret;
- down(&hr_dev->cmd.poll_sem);
+ mutex_lock(&hr_dev->cmd.poll_mutex);
ret = __hns_roce_cmd_mbox_poll(hr_dev, in_param, out_param, in_modifier,
op_modifier, op, timeout);
- up(&hr_dev->cmd.poll_sem);
+ mutex_unlock(&hr_dev->cmd.poll_mutex);
return ret;
}
@@ -275,7 +275,7 @@ int hns_roce_cmd_init(struct hns_roce_dev *hr_dev)
struct device *dev = &hr_dev->pdev->dev;
mutex_init(&hr_dev->cmd.hcr_mutex);
- sema_init(&hr_dev->cmd.poll_sem, 1);
+ mutex_init(&hr_dev->cmd.poll_mutex);
hr_dev->cmd.use_events = 0;
hr_dev->cmd.toggle = 1;
hr_dev->cmd.max_cmds = CMD_MAX_NUM;
@@ -319,8 +319,6 @@ int hns_roce_cmd_use_events(struct hns_roce_dev *hr_dev)
hr_cmd->token_mask = CMD_TOKEN_MASK;
hr_cmd->use_events = 1;
- down(&hr_cmd->poll_sem);
-
return 0;
}
@@ -335,7 +333,6 @@ void hns_roce_cmd_use_polling(struct hns_roce_dev *hr_dev)
down(&hr_cmd->event_sem);
kfree(hr_cmd->context);
- up(&hr_cmd->poll_sem);
}
struct hns_roce_cmd_mailbox
diff --git a/drivers/infiniband/hw/hns/hns_roce_device.h b/drivers/infiniband/hw/hns/hns_roce_device.h
index 3417315..2afe075 100644
--- a/drivers/infiniband/hw/hns/hns_roce_device.h
+++ b/drivers/infiniband/hw/hns/hns_roce_device.h
@@ -34,6 +34,7 @@
#define _HNS_ROCE_DEVICE_H
#include <rdma/ib_verbs.h>
+#include <linux/mutex.h>
#define DRV_NAME "hns_roce"
@@ -358,7 +359,7 @@ struct hns_roce_cmdq {
struct dma_pool *pool;
u8 __iomem *hcr;
struct mutex hcr_mutex;
- struct semaphore poll_sem;
+ struct mutex poll_mutex;
/*
* Event mode: cmd register mutex protection,
* ensure to not exceed max_cmds and user use limit region
--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
^ permalink raw reply related
* [PATCH v2 2/8] IB/core: Replace semaphore sm_sem with completion
From: Binoy Jayan @ 2016-10-25 12:01 UTC (permalink / raw)
To: Doug Ledford, Sean Hefty, Hal Rosenstock
Cc: Arnd Bergmann, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA, Binoy Jayan
In-Reply-To: <1477396919-27669-1-git-send-email-binoy.jayan-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
The semaphore 'sm_sem' is used as completion, so convert it to
struct completion. Semaphores are going away in the future. The initial
status of the completion variable is marked as completed by a call to
the function 'complete' immediately following the initialization.
Signed-off-by: Binoy Jayan <binoy.jayan-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
---
drivers/infiniband/core/user_mad.c | 14 ++++++++------
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/drivers/infiniband/core/user_mad.c b/drivers/infiniband/core/user_mad.c
index 415a318..df070cc 100644
--- a/drivers/infiniband/core/user_mad.c
+++ b/drivers/infiniband/core/user_mad.c
@@ -47,6 +47,7 @@
#include <linux/kref.h>
#include <linux/compat.h>
#include <linux/sched.h>
+#include <linux/completion.h>
#include <linux/semaphore.h>
#include <linux/slab.h>
@@ -87,7 +88,7 @@ struct ib_umad_port {
struct cdev sm_cdev;
struct device *sm_dev;
- struct semaphore sm_sem;
+ struct completion sm_comp;
struct mutex file_mutex;
struct list_head file_list;
@@ -1030,12 +1031,12 @@ static int ib_umad_sm_open(struct inode *inode, struct file *filp)
port = container_of(inode->i_cdev, struct ib_umad_port, sm_cdev);
if (filp->f_flags & O_NONBLOCK) {
- if (down_trylock(&port->sm_sem)) {
+ if (!try_wait_for_completion(&port->sm_comp)) {
ret = -EAGAIN;
goto fail;
}
} else {
- if (down_interruptible(&port->sm_sem)) {
+ if (wait_for_completion_interruptible(&port->sm_comp)) {
ret = -ERESTARTSYS;
goto fail;
}
@@ -1060,7 +1061,7 @@ static int ib_umad_sm_open(struct inode *inode, struct file *filp)
ib_modify_port(port->ib_dev, port->port_num, 0, &props);
err_up_sem:
- up(&port->sm_sem);
+ complete(&port->sm_comp);
fail:
return ret;
@@ -1079,7 +1080,7 @@ static int ib_umad_sm_close(struct inode *inode, struct file *filp)
ret = ib_modify_port(port->ib_dev, port->port_num, 0, &props);
mutex_unlock(&port->file_mutex);
- up(&port->sm_sem);
+ complete(&port->sm_comp);
kobject_put(&port->umad_dev->kobj);
@@ -1177,7 +1178,8 @@ static int ib_umad_init_port(struct ib_device *device, int port_num,
port->ib_dev = device;
port->port_num = port_num;
- sema_init(&port->sm_sem, 1);
+ init_completion(&port->sm_comp);
+ complete(&port->sm_comp);
mutex_init(&port->file_mutex);
INIT_LIST_HEAD(&port->file_list);
--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH v2 1/8] IB/core: iwpm_nlmsg_request: Replace semaphore with completion
From: Binoy Jayan @ 2016-10-25 12:01 UTC (permalink / raw)
To: Doug Ledford, Sean Hefty, Hal Rosenstock
Cc: Arnd Bergmann, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA, Binoy Jayan
In-Reply-To: <1477396919-27669-1-git-send-email-binoy.jayan-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
Semaphore sem in iwpm_nlmsg_request is used as completion, so
convert it to a struct completion type. Semaphores are going
away in the future.
Signed-off-by: Binoy Jayan <binoy.jayan-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
---
drivers/infiniband/core/iwpm_msg.c | 8 ++++----
drivers/infiniband/core/iwpm_util.c | 7 +++----
drivers/infiniband/core/iwpm_util.h | 3 ++-
3 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/drivers/infiniband/core/iwpm_msg.c b/drivers/infiniband/core/iwpm_msg.c
index 1c41b95..761358f 100644
--- a/drivers/infiniband/core/iwpm_msg.c
+++ b/drivers/infiniband/core/iwpm_msg.c
@@ -394,7 +394,7 @@ int iwpm_register_pid_cb(struct sk_buff *skb, struct netlink_callback *cb)
/* always for found nlmsg_request */
kref_put(&nlmsg_request->kref, iwpm_free_nlmsg_request);
barrier();
- up(&nlmsg_request->sem);
+ complete(&nlmsg_request->comp);
return 0;
}
EXPORT_SYMBOL(iwpm_register_pid_cb);
@@ -463,7 +463,7 @@ int iwpm_add_mapping_cb(struct sk_buff *skb, struct netlink_callback *cb)
/* always for found request */
kref_put(&nlmsg_request->kref, iwpm_free_nlmsg_request);
barrier();
- up(&nlmsg_request->sem);
+ complete(&nlmsg_request->comp);
return 0;
}
EXPORT_SYMBOL(iwpm_add_mapping_cb);
@@ -555,7 +555,7 @@ int iwpm_add_and_query_mapping_cb(struct sk_buff *skb,
/* always for found request */
kref_put(&nlmsg_request->kref, iwpm_free_nlmsg_request);
barrier();
- up(&nlmsg_request->sem);
+ complete(&nlmsg_request->comp);
return 0;
}
EXPORT_SYMBOL(iwpm_add_and_query_mapping_cb);
@@ -749,7 +749,7 @@ int iwpm_mapping_error_cb(struct sk_buff *skb, struct netlink_callback *cb)
/* always for found request */
kref_put(&nlmsg_request->kref, iwpm_free_nlmsg_request);
barrier();
- up(&nlmsg_request->sem);
+ complete(&nlmsg_request->comp);
return 0;
}
EXPORT_SYMBOL(iwpm_mapping_error_cb);
diff --git a/drivers/infiniband/core/iwpm_util.c b/drivers/infiniband/core/iwpm_util.c
index ade71e7..08ddd2e 100644
--- a/drivers/infiniband/core/iwpm_util.c
+++ b/drivers/infiniband/core/iwpm_util.c
@@ -323,8 +323,7 @@ struct iwpm_nlmsg_request *iwpm_get_nlmsg_request(__u32 nlmsg_seq,
nlmsg_request->nl_client = nl_client;
nlmsg_request->request_done = 0;
nlmsg_request->err_code = 0;
- sema_init(&nlmsg_request->sem, 1);
- down(&nlmsg_request->sem);
+ init_completion(&nlmsg_request->comp);
return nlmsg_request;
}
@@ -368,8 +367,8 @@ int iwpm_wait_complete_req(struct iwpm_nlmsg_request *nlmsg_request)
{
int ret;
- ret = down_timeout(&nlmsg_request->sem, IWPM_NL_TIMEOUT);
- if (ret) {
+ ret = wait_for_completion_timeout(&nlmsg_request->comp, IWPM_NL_TIMEOUT);
+ if (!ret) {
ret = -EINVAL;
pr_info("%s: Timeout %d sec for netlink request (seq = %u)\n",
__func__, (IWPM_NL_TIMEOUT/HZ), nlmsg_request->nlmsg_seq);
diff --git a/drivers/infiniband/core/iwpm_util.h b/drivers/infiniband/core/iwpm_util.h
index af1fc14..ea6c299 100644
--- a/drivers/infiniband/core/iwpm_util.h
+++ b/drivers/infiniband/core/iwpm_util.h
@@ -43,6 +43,7 @@
#include <linux/delay.h>
#include <linux/workqueue.h>
#include <linux/mutex.h>
+#include <linux/completion.h>
#include <linux/jhash.h>
#include <linux/kref.h>
#include <net/netlink.h>
@@ -69,7 +70,7 @@ struct iwpm_nlmsg_request {
u8 nl_client;
u8 request_done;
u16 err_code;
- struct semaphore sem;
+ struct completion comp;
struct kref kref;
};
--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH v2 0/8] infiniband: Remove semaphores
From: Binoy Jayan @ 2016-10-25 12:01 UTC (permalink / raw)
To: Doug Ledford, Sean Hefty, Hal Rosenstock
Cc: Arnd Bergmann, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA, Binoy Jayan
Hi,
These are a set of patches [v2] which removes semaphores from infiniband.
These are part of a bigger effort to eliminate all semaphores from the
linux kernel.
v1 -> v2:
IB/hns : Use wait_event instead of open coding counting semaphores
IB/mthca : Use wait_event instead of open coding counting semaphores
IB/mthca : Remove mutex_[un]lock from *_cmd_use_events/*_cmd_use_polling
IB/mlx5 : Cleanup, add helper mlx5_ib_post_send_wait
Thanks,
Binoy
Binoy Jayan (8):
IB/core: iwpm_nlmsg_request: Replace semaphore with completion
IB/core: Replace semaphore sm_sem with completion
IB/hns: Replace semaphore poll_sem with mutex
IB/mthca: Replace semaphore poll_sem with mutex
IB/isert: Replace semaphore sem with completion
IB/hns: Replace counting semaphore event_sem with wait_event
IB/mthca: Replace counting semaphore event_sem with wait_event
IB/mlx5: Add helper mlx5_ib_post_send_wait
drivers/infiniband/core/iwpm_msg.c | 8 +--
drivers/infiniband/core/iwpm_util.c | 7 +--
drivers/infiniband/core/iwpm_util.h | 3 +-
drivers/infiniband/core/user_mad.c | 14 +++--
drivers/infiniband/hw/hns/hns_roce_cmd.c | 48 ++++++++++-----
drivers/infiniband/hw/hns/hns_roce_device.h | 5 +-
drivers/infiniband/hw/mlx5/mr.c | 96 +++++++++--------------------
drivers/infiniband/hw/mthca/mthca_cmd.c | 47 +++++++++-----
drivers/infiniband/hw/mthca/mthca_cmd.h | 1 +
drivers/infiniband/hw/mthca/mthca_dev.h | 5 +-
drivers/infiniband/ulp/isert/ib_isert.c | 6 +-
drivers/infiniband/ulp/isert/ib_isert.h | 3 +-
12 files changed, 119 insertions(+), 124 deletions(-)
--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 1/3] memremap.c : Add support for ZONE_DEVICE IO memory with struct pages.
From: Stephen Bates @ 2016-10-25 11:54 UTC (permalink / raw)
To: Dan Williams
Cc: linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org,
linux-rdma, linux-block, Linux MM, Ross Zwisler, Matthew Wilcox,
Jason Gunthorpe, haggaie, Christoph Hellwig, Jens Axboe,
Jonathan Corbet, jim.macdonald, sbates, Logan Gunthorpe
In-Reply-To: <CAPcyv4iTFpZ1b74Wf+qWe9=Annp+-OCy+pFMS7Fo7quUFwhM4g@mail.gmail.com>
On Wed, Oct 19, 2016 at 01:01:06PM -0700, Dan Williams wrote:
> >>
> >> In the cover letter, "[PATCH 0/3] iopmem : A block device for PCIe
> >> memory", it mentions that the lack of I/O coherency is a known issue
> >> and users of this functionality need to be cognizant of the pitfalls.
> >> If that is the case why do we need support for different cpu mapping
> >> types than the default write-back cache setting? It's up to the
> >> application to handle cache cpu flushing similar to what we require of
> >> device-dax users in the persistent memory case.
> >
> > Some of the iopmem hardware we have tested has certain alignment
> > restrictions on BAR accesses. At the very least we require write
> > combine mappings for these. We then felt it appropriate to add the
> > other mappings for the sake of completeness.
>
> If the device can support write-combine then it can support bursts, so
> I wonder why it couldn't support read bursts for cache fills...
Dan
You make a good point. We did some testing on this and for the HW we
have access too we did see that a standard WB mapping worked.
Interestly though the local access performance was much slower than
for the WC mapping. We also noticed the PAT entries we not marked
correctly in the WB case. I am trying to get access to some other HW
for more testing.
Grepping for the address of interest:
In WB mode it's:
uncached-minus @ 0x381f80000000-0x381fc0000000
In WC mode it's:
write-combining @ 0x381f80000000-0x381fc0000000
Cheers
Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH 0/3] iopmem : A block device for PCIe memory
From: Stephen Bates @ 2016-10-25 11:50 UTC (permalink / raw)
To: Dave Chinner
Cc: Christoph Hellwig, Dan Williams, linux-kernel@vger.kernel.org,
linux-nvdimm@lists.01.org, linux-rdma, linux-block, Linux MM,
Ross Zwisler, Matthew Wilcox, jgunthorpe, haggaie, Jens Axboe,
Jonathan Corbet, jim.macdonald, sbates, Logan Gunthorpe,
David Woodhouse, Raj, Ashok
In-Reply-To: <20161021111253.GQ14023@dastard>
Hi Dave and Christoph
On Fri, Oct 21, 2016 at 10:12:53PM +1100, Dave Chinner wrote:
> On Fri, Oct 21, 2016 at 02:57:14AM -0700, Christoph Hellwig wrote:
> > On Fri, Oct 21, 2016 at 10:22:39AM +1100, Dave Chinner wrote:
> > > You do realise that local filesystems can silently change the
> > > location of file data at any point in time, so there is no such
> > > thing as a "stable mapping" of file data to block device addresses
> > > in userspace?
> > >
> > > If you want remote access to the blocks owned and controlled by a
> > > filesystem, then you need to use a filesystem with a remote locking
> > > mechanism to allow co-ordinated, coherent access to the data in
> > > those blocks. Anything else is just asking for ongoing, unfixable
> > > filesystem corruption or data leakage problems (i.e. security
> > > issues).
> >
Dave are you saying that even for local mappings of files on a DAX
capable system it is possible for the mappings to move on you unless
the FS supports locking? Does that not mean DAX on such FS is
inherently broken?
> > And at least for XFS we have such a mechanism :) E.g. I have a
> > prototype of a pNFS layout that uses XFS+DAX to allow clients to do
> > RDMA directly to XFS files, with the same locking mechanism we use
> > for the current block and scsi layout in xfs_pnfs.c.
>
Thanks for fixing this issue on XFS Christoph! I assume this problem
continues to exist on the other DAX capable FS?
One more reason to consider a move to /dev/dax I guess ;-)...
Stephen
> Oh, that's good to know - pNFS over XFS was exactly what I was
> thinking of when I wrote my earlier reply. A few months ago someone
> else was trying to use file mappings in userspace for direct remote
> client access on fabric connected devices. I told them "pNFS on XFS
> and write an efficient transport for you hardware"....
>
> Now that I know we've got RDMA support for pNFS on XFS in the
> pipeline, I can just tell them "just write an rdma driver for your
> hardware" instead. :P
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox