[PATCH net-next V2 00/21] net/mlx4: SRIOV support

[PATCH net-next V2 00/21] net/mlx4: SRIOV support

[Yevgeny Petrilin]

Hello David,

This is V2 of a patch set that adds SRIOV support for ConnectX2 and ConnectX3 HW for Ethernet Link layer.
Infiniband and RoCE SRIOV support will be added at later stage

In ConnectX, nearly all device resources can be associated with any function.
Once allocated, these resources can be manipulated exclusively by their
associated functions in the same manner as in the single-function case.
The same driver acts both as pf and vf driver.
mlx4_en module (Ethernet specific code) remains virtualization unaware and all the multifunction logic is
implemented in mlx4_core module.
To keep that separation, some code (mainly port management) moved from mlx4_en to mlx4_core module.

The device continues to use a single HCA Command Register (HCR), which is owned by the pf.
Thus, the pf is given the role of allocating resources to vfs, and controlling shared resource (such as ports).

A dedicated HW communication channel is used to transfer resource allocation and
modification requests from vfs to the pf, and events from the pf to the vfs.
Consequently, no hypervisor-specific services are required for pf-vf communication.
All vfs resources are tracked by the pf and cleaned up in case the vf doesn't close gracefully.
The resource tracker also prevents untrusted vfs to manipulate resources that belong to other functions.

---
Change from V1:
 - Fixed warnings found by -Wswitch-enum compilation flag.
Change from V0:
- Use enum {MLX4_CMD_NATIVE, MLX4_CMD_WRAPPED} instead of 1/0 when calling mlx4_cmd
- Added pf context behavior to mlx4_func_cap and query func cap
- Removed ifdef CONFIG_PCI_IOV, all modules parameters do not depend on this define anymore
- sr_iov module parameter changed to num_vfs
- made probe_vf parameter writable, allows binding of VF back to hypervisor

mlx4_en: updated driver version to 2.0
mlx4_core: updated driver version to 1.1
mlx4_core: Modify driver initialization flow to accommodate SRIOV for Ethernet.
mlx4_core: adjust catas operation for SRIOV mode.
mlx4_core: mtts resources units changed to offset
mlx4_en: Allow communication between functions on same host
mlx4: Ethernet port management modifications
mlx4: Traffic steering management support for SRIOV
mlx4_ib: disable SRIOV mode for IB ports (not yet supported).
mlx4_core:  resource tracking for HCA resources used by guests.
mlx4_core: Add wrapper functions and comm channel and slave event support to EQs
mlx4_core: mtt modifications for SRIOV
mlx4_core: cq modifications for SRIOV
mlx4_core: qp modifications for SRIOV
mlx4_core: srq modifications for SRIOV
mlx4_core: Added FW commands and their wrappers for supporting SRIOV.
mlx4_core: Implement the master-slave communication channel.
mlx4_core: Reduce number of PD bits to 17.
mlx4_core:  Add "native" argument to mlx4_cmd and its callers (where needed)
mlx4: Extanding port_mask functionality
mlx4_core:  initial header-file changes for SRIOV support.

 drivers/infiniband/hw/mlx4/mad.c                      |    6
 drivers/infiniband/hw/mlx4/main.c                     |   14
 drivers/net/ethernet/mellanox/mlx4/Makefile           |    2
 drivers/net/ethernet/mellanox/mlx4/catas.c            |    7
 drivers/net/ethernet/mellanox/mlx4/cmd.c              | 1308 +++++++
 drivers/net/ethernet/mellanox/mlx4/cq.c               |  141
 drivers/net/ethernet/mellanox/mlx4/en_netdev.c        |   32
 drivers/net/ethernet/mellanox/mlx4/en_port.c          |   84
 drivers/net/ethernet/mellanox/mlx4/en_port.h          |   43
 drivers/net/ethernet/mellanox/mlx4/en_rx.c            |   15
 drivers/net/ethernet/mellanox/mlx4/en_selftest.c      |    2
 drivers/net/ethernet/mellanox/mlx4/en_tx.c            |   20
 drivers/net/ethernet/mellanox/mlx4/eq.c               |  430 +-
 drivers/net/ethernet/mellanox/mlx4/fw.c               |  404 ++
 drivers/net/ethernet/mellanox/mlx4/fw.h               |   26
 drivers/net/ethernet/mellanox/mlx4/icm.c              |    5
 drivers/net/ethernet/mellanox/mlx4/intf.c             |    6
 drivers/net/ethernet/mellanox/mlx4/main.c             |  868 ++++-
 drivers/net/ethernet/mellanox/mlx4/mcg.c              |  228 +
 drivers/net/ethernet/mellanox/mlx4/mlx4.h             |  670 +++
 drivers/net/ethernet/mellanox/mlx4/mlx4_en.h          |    4
 drivers/net/ethernet/mellanox/mlx4/mr.c               |  486 ++
 drivers/net/ethernet/mellanox/mlx4/pd.c               |   19
 drivers/net/ethernet/mellanox/mlx4/port.c             |  616 ++-
 drivers/net/ethernet/mellanox/mlx4/profile.c          |    9
 drivers/net/ethernet/mellanox/mlx4/qp.c               |  238 +
 drivers/net/ethernet/mellanox/mlx4/resource_tracker.c | 3103 ++++++++++++++++++
 drivers/net/ethernet/mellanox/mlx4/sense.c            |    3
 drivers/net/ethernet/mellanox/mlx4/srq.c              |  132
 include/linux/mlx4/cmd.h                              |   51
 include/linux/mlx4/device.h                           |   72
 31 files changed, 8119 insertions(+), 925 deletions(-)


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH net-next V2 00/21] net/mlx4: SRIOV support

[David Miller]

From: Yevgeny Petrilin <yevgenyp-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
Date: Tue, 13 Dec 2011 16:10:25 +0200

> This is V2 of a patch set that adds SRIOV support for ConnectX2 and
> ConnectX3 HW for Ethernet Link layer.  Infiniband and RoCE SRIOV
> support will be added at later stage

Applied.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH net-next V2 00/21] net/mlx4: SRIOV support

[Yevgeny Petrilin]

> > This is V2 of a patch set that adds SRIOV support for ConnectX2 and
> > ConnectX3 HW for Ethernet Link layer.  Infiniband and RoCE SRIOV
> > support will be added at later stage
> 
> Applied.

Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH net-next V2 00/21] net/mlx4: SRIOV support

[Yinghai Lu]

On Tue, Dec 13, 2011 at 11:16 AM, Yevgeny Petrilin
<yevgenyp-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>> > This is V2 of a patch set that adds SRIOV support for ConnectX2 and
>> > ConnectX3 HW for Ethernet Link layer.  Infiniband and RoCE SRIOV
>> > support will be added at later stage
>>

got following warning:

drivers/built-in.o: In function `mlx4_en_set_pauseparam':
/home/yhlu/xx/xx/kernel/tip/linux-2.6/drivers/net/ethernet/mellanox/mlx4/en_ethtool.c:387:
undefined reference to `mlx4_SET_PORT_general'
drivers/built-in.o: In function `mlx4_en_do_set_multicast':
/home/yhlu/xx/xx/kernel/tip/linux-2.6/drivers/net/ethernet/mellanox/mlx4/en_netdev.c:233:
undefined reference to `mlx4_SET_PORT_qpn_calc'
/home/yhlu/xx/xx/kernel/tip/linux-2.6/drivers/net/ethernet/mellanox/mlx4/en_netdev.c:277:
undefined reference to `mlx4_SET_PORT_qpn_calc'
drivers/built-in.o: In function `mlx4_en_start_port':
/home/yhlu/xx/xx/kernel/tip/linux-2.6/drivers/net/ethernet/mellanox/mlx4/en_netdev.c:664:
undefined reference to `mlx4_SET_PORT_general'
/home/yhlu/xx/xx/kernel/tip/linux-2.6/drivers/net/ethernet/mellanox/mlx4/en_netdev.c:676:
undefined reference to `mlx4_SET_PORT_qpn_calc'
drivers/built-in.o: In function `mlx4_en_init_netdev':
/home/yhlu/xx/xx/kernel/tip/linux-2.6/drivers/net/ethernet/mellanox/mlx4/en_netdev.c:1138:
undefined reference to `mlx4_SET_PORT_general'
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH net-next V2 00/21] net/mlx4: SRIOV support

[Or Gerlitz]

On Wed, Dec 14, 2011 at 10:42 PM, Yinghai Lu <yinghai-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:

> got following warning:

could you send your .config as attached to Yevgeny Petrilin
<yevgenyp-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, he will distribute it to the group

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH net-next V2 00/21] net/mlx4: SRIOV support

[Yinghai Lu]

On Wed, Dec 14, 2011 at 1:50 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Wed, Dec 14, 2011 at 10:42 PM, Yinghai Lu <yinghai-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>
>> got following warning:
>
> could you send your .config as attached to Yevgeny Petrilin
> <yevgenyp-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, he will distribute it to the group

hotplug removal is broken...

sca21d-0a86d3dd:~ # echo 0 > /sys/bus/pci/slots/1/power
[  123.085198] pciehp 0000:00:02.2:pcie04: disable_slot: physical_slot = 1
[  123.091863] pciehp 0000:00:02.2:pcie04: pciehp_get_power_status:
SLOTCTRL a8 value read 1f9
[  123.100210] pciehp 0000:00:02.2:pcie04: pciehp_unconfigure_device:
domain:bus:dev = 0000:30:00
[  123.114078] BUG: unable to handle kernel NULL pointer dereference
at 0000000000000010
[  123.121923] IP: [<ffffffff8177fc92>] mlx4_table_put+0x57/0xaf
[  123.127684] PGD 0
[  123.129712] Oops: 0000 [#1] SMP
[  123.132969] CPU 0
[  123.134806] Modules linked in:
[  123.138063]
[  123.139552] Pid: 7617, comm: bash Not tainted
3.2.0-rc7-tip-yh-03751-ga431a72-dirty #35 Oracle Corporation Sun Blade
/BD,ASSY,
[  123.151662] RIP: 0010:[<ffffffff8177fc92>]  [<ffffffff8177fc92>]
mlx4_table_put+0x57/0xaf
[  123.159850] RSP: 0018:ffff8802707018d8  EFLAGS: 00010296
[  123.165151] RAX: ffff880175a69900 RBX: ffff8801756f4f50 RCX: ffff880276c407e8
[  123.172272] RDX: 0000000000000000 RSI: ffff880276c407e8 RDI: ffff880270701898
[  123.179391] RBP: ffff880270701908 R08: ffff880270701878 R09: 3d605d874c4567fa
[  123.186512] R10: 3d605d874c4567e5 R11: dead000000200200 R12: 0000000000000000
[  123.193633] R13: ffff8801756f4f70 R14: ffff8801756f0000 R15: 0000000000000000
[  123.200754] FS:  00007f44d66686f0(0000) GS:ffff880179e00000(0000)
knlGS:0000000000000000
[  123.208827] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  123.214564] CR2: 0000000000000010 CR3: 000000016bf94000 CR4: 00000000000407f0
[  123.221684] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  123.228805] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  123.235926] Process bash (pid: 7617, threadinfo ffff880270700000,
task ffff880276c40000)
[  123.243997] Stack:
[  123.246009]  0000000000001dbf ffff8801756f4f50 ffff8801756f0000
0000000000001d80
[  123.253458]  0000000000040000 0000000000001dbf ffff880270701948
ffffffff8177fec7
[  123.260908]  0000000000000040 0000000000001d80 0000000000000006
ffff8801756f0000
[  123.268357] Call Trace:
[  123.270808]  [<ffffffff8177fec7>] mlx4_table_put_range+0x35/0x50
[  123.276806]  [<ffffffff817857f7>] __mlx4_free_mtt_range+0x5c/0x65
[  123.282893]  [<ffffffff81785887>] mlx4_free_mtt_range+0x87/0x92
[  123.288802]  [<ffffffff81779d4e>] ? __mlx4_cmd+0x4a/0x6b
[  123.294105]  [<ffffffff817854e9>] ? mlx4_HW2SW_MPT.constprop.22+0x43/0x45
[  123.300878]  [<ffffffff817858a9>] mlx4_mtt_cleanup+0x17/0x19
[  123.306527]  [<ffffffff8178590c>] mlx4_mr_free_reserved+0x61/0x66
[  123.312607]  [<ffffffff81785d6a>] mlx4_mr_free+0x17/0x3e
[  123.317910]  [<ffffffff81785db0>] mlx4_fmr_free+0x1f/0x2c
[  123.323307]  [<ffffffff81a32e3e>] mlx4_ib_fmr_dealloc+0x24/0x3e
[  123.329215]  [<ffffffff81134437>] ? kfree+0x125/0x131
[  123.334257]  [<ffffffff81981fb7>] ib_dealloc_fmr+0x18/0x25
[  123.339732]  [<ffffffff81984e56>] ib_destroy_fmr_pool+0x8c/0xf7
[  123.345640]  [<ffffffff81a5ebc7>] srp_remove_one+0x19d/0x1c6
[  123.351287]  [<ffffffff81984429>] ib_unregister_device+0x3d/0xe3
[  123.357284]  [<ffffffff81a30058>] mlx4_ib_remove+0x26/0xed
[  123.362756]  [<ffffffff81780246>] mlx4_remove_device+0x57/0x72
[  123.368577]  [<ffffffff8178052a>] mlx4_unregister_device+0x3c/0x72
[  123.374744]  [<ffffffff81782477>] mlx4_remove_one+0xb9/0x23f
[  123.380395]  [<ffffffff81359137>] pci_device_remove+0x3c/0xbc
[  123.386129]  [<ffffffff81416ee7>] __device_release_driver+0x86/0xd2
[  123.392383]  [<ffffffff81416f58>] device_release_driver+0x25/0x32
[  123.398464]  [<ffffffff81416aee>] bus_remove_device+0x93/0xa3
[  123.404199]  [<ffffffff81414d35>] device_del+0x12d/0x17f
[  123.409499]  [<ffffffff81414d9a>] device_unregister+0x13/0x1f
[  123.415235]  [<ffffffff81354826>] pci_stop_bus_device+0x8a/0xad
[  123.421142]  [<ffffffff81354984>] pci_remove_bus_device+0x13/0x1f
[  123.427226]  [<ffffffff81366ac6>] pciehp_unconfigure_device+0x107/0x16d
[  123.433825]  [<ffffffff81366539>] pciehp_disable_slot+0x11e/0x186
[  123.439906]  [<ffffffff813667ec>] pciehp_sysfs_disable_slot+0x68/0x108
[  123.446420]  [<ffffffff81365929>] disable_slot+0x58/0x5c
[  123.451722]  [<ffffffff8136273e>] power_write_file+0xd1/0x11b
[  123.457457]  [<ffffffff8136266d>] ? get_attention_status+0xad/0xad
[  123.463625]  [<ffffffff8135c9b9>] pci_slot_attr_store+0x29/0x2b
[  123.469533]  [<ffffffff811960f8>] sysfs_write_file+0x10e/0x14a
[  123.475354]  [<ffffffff8113b2cd>] vfs_write+0xb5/0x128
[  123.480483]  [<ffffffff8113b529>] sys_write+0x4d/0x77
[  123.485526]  [<ffffffff81c653d2>] system_call_fastpath+0x16/0x1b
[  123.491517] Code: 89 ef 41 21 c7 89 c8 99 f7 7e 10 31 f6 89 c1 44
89 f8 99 f7 f9 41 89 c7 e8 e8 c0 4d 00 48 8b 83 c8 00 00 00 4d 63 e7
4a 8b 14 e0 <8b> 42 10 ff c8 85 c0 89 42 10 75 37 41 c1 e7 12 4c 89 f7
49 63
[  123.511545] RIP  [<ffffffff8177fc92>] mlx4_table_put+0x57/0xaf
[  123.517383]  RSP <ffff8802707018d8>
[  123.520866] CR2: 0000000000000010
[  123.524228] ---[ end trace 6436e31aa4627ed0 ]---
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH net-next V2 00/21] net/mlx4: SRIOV support

[Roland Dreier]

On Fri, Dec 30, 2011 at 2:23 PM, Yinghai Lu <yinghai-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:

> hotplug removal is broken...

Is this a new regression with these patches?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH net-next V2 00/21] net/mlx4: SRIOV support

[Yinghai Lu]

On Fri, Dec 30, 2011 at 9:57 PM, Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
> On Fri, Dec 30, 2011 at 2:23 PM, Yinghai Lu <yinghai-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>
>> hotplug removal is broken...
>
> Is this a new regression with these patches?
>

yes.

after reverting those patches, removal work well without problem.

Thanks

Yinghai
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH net-next V2 00/21] net/mlx4: SRIOV support

[Yevgeny Petrilin]

> >
> >> hotplug removal is broken...
> >
> > Is this a new regression with these patches?
> >
> 
> yes.
> 
> after reverting those patches, removal work well without problem.
> 
> Thanks
> 
> Yinghai

Thanks,
We just found the same failure internally.
Will send a fix shortly.

Yevgeny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html