linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 00/22] NTB: Alter kernel API to support multi-port devices
@ 2016-11-29 17:15 Serge Semin
  2016-11-29 17:15 ` [PATCH 01/22] NTB: Move link state API being first in sources Serge Semin
                   ` (22 more replies)
  0 siblings, 23 replies; 62+ messages in thread
From: Serge Semin @ 2016-11-29 17:15 UTC (permalink / raw)
  To: jdmason, dave.jiang, Allen.Hubbe, Xiangliang.Yu
  Cc: Sergey.Semin, linux-ntb, linux-kernel, Serge Semin

There are devices, like IDT PCIe-switches, which have more than just two ports.
Particularly one device can have up to eight ports with NTB-function activated.
In order to support such devices, NTB kernel API should be altered since
currently it's optimized to work with two-ports devices only.

Here are the changes I made to conform the design we discussed a few months ago:

1) Port-index-related methods are added to KAPI
 ntb_port_number();
 ntb_peer_port_count();
 ntb_peer_port_number(pdix);
 ntb_peer_port_idx(port);

2) Link state method returns bitfield of link states for each reachable port
 u64 ntb_link_is_up();
 
3) Link enable/disable methods work with local link side of NTB
 ntb_link_enable()/ntb_link_disable();
 
4) NTB memory window related interface does the following things
 ntb_mw_count(pidx); - number of inbound memory windows, which can be allocated
for shared buffer with specified peer device.
 ntb_mw_get_align(pidx, widx); - get alignment and size restrition parameters
to properly allocate inbound memory region.
 ntb_peer_mw_count(); - get number of outbound memory windows.
 ntb_peer_mw_get_addr(widx); - get mapping address of an outbound memory window 

Inbound MW based hardware:
 ntb_mw_set_trans(pidx, widx); - set translation address of allocated inbound
memory window so a peer device could access it.
 ntb_mw_clear_trans(pidx, widx); - clear the translation address of an inbound
memory window.

Outbound MW based hardware:
 ntb_peer_mw_set_trans(pidx, widx); - set translation address of a memory
window retrieved from a peer device
 ntb_peer_mw_clear_trans(pidx, widx); - clear the translation address of an
outbound memory window

5) Scratchpad interface needs to support multi-port devices as well
 ntb_spad_count() - return number of Scratchpad per each port
 ntb_peer_spad_addr(pidx, sidx) - address of Scratchpad register of the
peer device with pidx-index
 ntb_peer_spad_read(pidx, sidx) - read specified Scratchpad register of the
peer with pidx-index
 ntb_peer_spad_write(pidx, sidx) - write data to Scratchpad register of the
peer with pidx-index

6) Introduce new messaging interface of NTB KAPI
 ntb_msg_count(); - get number of message registers
 ntb_msg_inbits(); - get bitfield of inbound message registers status
 ntb_msg_outbits(); - get bitfield of outbound message registers status
 ntb_msg_read_sts(); - read the inbound and outbound message registers status
 ntb_msg_clear_sts(); - clear status bits of message registers
 ntb_msg_set_mask(); - mask interrupts raised by status bits of message
registers.
 ntb_msg_clear_mask(); - clear interrupts mask bits of message registers
 ntb_msg_recv(midx, *pidx); - read message register with specified index,
additionally getting peer port index which data received from
 ntb_msg_send(midx, pidx); - write data to the specified message register
sending it to the passed peer device connected over a pidx port
 ntb_msg_event(); - notify driver context of a new message event
 
7) Topology reduced to be either P2P (port-to-port) or B2B (bridge-to-bridge).
Since there is port number introduced to be part of ntb_dev structure, real
port number can be used to determine Primary and Secondary sides. Intel and AMD
driver are altered to support this novation.

8) Standard test drivers: PingPong, Debugging tool and Raw Perf as well as NTB
Transport drivers don't support multi-port devices at the moment.
Since we haven't got any real multi-port hadrware driver, it's dangerous to
make any serious alterations in the debugging tools. So I have made those
drivers to work the way they wokred before, but using the new NTB API.
The situation will change when I finish porting my current IDT NTB driver to
support new API. Then I'll be able to refactor the tools and test them using
real multi-port hardware.

The changes are made on top of the NTB-fork of the kernel:
https://github.com/jonmason/ntb
of "ntb"-branch with last commit:
9c763584b7c8911106bb77af7e648bef09af9d80 Linux 4.9-rc6

Signed-off-by: Serge Semin <fancer.lancer@gmail.com>

Serge Semin (22):
  NTB: Move link state API being first in sources
  NTB: Add peer indexed ports NTB API
  NTB: Alter NTB API to support both inbound and outbound MW based
    interfaces
  NTB: Add messaging NTB API
  NTB: Alter Scratchpads NTB API to support multi-ports interface
  NTB: Slightly alter link state NTB API
  NTB: Fix a few ntb.h issues
  NTB: Add T-Platforms copyrights to NTB API
  NTB Intel: Move link-related methods being first in the driver
  NTB Intel: Add port-related NTB API callback methods
  NTB Intel: Alter MW interface to fit new NTB API
  NTB Intel: Alter Scratchpads interface to fit new NTB API
  NTB Intel: Add T-Platforms copyrights to Intel NTB driver
  NTB AMD: Move link-related methods being first in the driver
  NTB AMD: Add port-related NTB API callback methods
  NTB AMD: Alter MW interface to fit new NTB API
  NTB AMD: Alter Scratchpads interface to fit new NTB API
  NTB AMD: Add T-Platforms copyrights to AMD NTB driver
  NTB PingPong: Alter driver to work with two-ports NTB API
  NTB Tool: Alter driver to work with two-ports NTB API
  NTB Perf: Alter driver to work with two-ports NTB API
  NTB Transport: Alter driver to work with two-ports NTB API

 drivers/ntb/hw/amd/ntb_hw_amd.c     | 310 +++++++++------
 drivers/ntb/hw/amd/ntb_hw_amd.h     |  12 +
 drivers/ntb/hw/intel/ntb_hw_intel.c | 420 ++++++++++++--------
 drivers/ntb/hw/intel/ntb_hw_intel.h |  12 +
 drivers/ntb/ntb.c                   |  15 +
 drivers/ntb/ntb_transport.c         |  43 +-
 drivers/ntb/test/ntb_perf.c         |  27 +-
 drivers/ntb/test/ntb_pingpong.c     |  11 +-
 drivers/ntb/test/ntb_tool.c         |  87 +++--
 include/linux/ntb.h                 | 753 +++++++++++++++++++++++++++++-------
 10 files changed, 1221 insertions(+), 469 deletions(-)

-- 
2.6.6

^ permalink raw reply	[flat|nested] 62+ messages in thread
* RE: [PATCH v2 7/9] NTB: Add new Memory Windows API documentation
@ 2016-12-13  4:14 Allen Hubbe
  0 siblings, 0 replies; 62+ messages in thread
From: Allen Hubbe @ 2016-12-13  4:14 UTC (permalink / raw)
  To: 'Serge Semin', jdmason, dave.jiang, Xiangliang.Yu
  Cc: Sergey.Semin, linux-ntb, linux-kernel

From: Serge Semin
> Since the new API slightly changes the way a typical NTB client driver
> works, the documentation file needs to be appropriately updated.
> 
> Signed-off-by: Serge Semin <fancer.lancer@gmail.com>

Acked-by: Allen Hubbe <Allen.Hubbe@dell.com>

> 
> ---
>  Documentation/ntb.txt | 99 ++++++++++++++++++++++++++++++++++++++++++++++-----
>  1 file changed, 91 insertions(+), 8 deletions(-)
> 
> diff --git a/Documentation/ntb.txt b/Documentation/ntb.txt
> index 1d9bbab..d01bb69 100644
> --- a/Documentation/ntb.txt
> +++ b/Documentation/ntb.txt
> @@ -1,14 +1,16 @@
>  # NTB Drivers
> 
>  NTB (Non-Transparent Bridge) is a type of PCI-Express bridge chip that connects
> -the separate memory systems of two computers to the same PCI-Express fabric.
> -Existing NTB hardware supports a common feature set, including scratchpad
> -registers, doorbell registers, and memory translation windows.  Scratchpad
> -registers are read-and-writable registers that are accessible from either side
> -of the device, so that peers can exchange a small amount of information at a
> -fixed address.  Doorbell registers provide a way for peers to send interrupt
> -events.  Memory windows allow translated read and write access to the peer
> -memory.
> +the separate memory systems of two or more computers to the same PCI-Express
> +fabric. Existing NTB hardware supports a common feature set: doorbell
> +registers and memory translation windows, as well as non common features like
> +scratchpad and message registers. Scratchpad registers are read-and-writable
> +registers that are accessible from either side of the device, so that peers can
> +exchange a small amount of information at a fixed address. Message registers can
> +be utialized for the same purpose. Additionally they are provided with with
> +special status bits to make sure the information isn't rewritten by another
> +peer. Doorbell registers provide a way for peers to send interrupt events.
> +Memory windows allow translated read and write access to the peer memory.
> 
>  ## NTB Core Driver (ntb)
> 
> @@ -26,6 +28,87 @@ as ntb hardware, or hardware drivers, are inserted and removed.  The
>  registration uses the Linux Device framework, so it should feel familiar to
>  anyone who has written a pci driver.
> 
> +### NTB Typical client driver implementation
> +
> +Primary purpose of NTB is to share some peace of memory between at least two
> +systems. So the NTB device features like Scratchpad/Message regiesters are
> +mainly used to perform the proper memory window initialization. Typically
> +there are two types of memory window interfaces supported by the NTB API:
> +inbound translation configured on the local ntb port and outbound translation
> +configured by the peer, on the peer ntb port. The first type is
> +depicted on the next figure
> +
> +Inbound translation:
> + Memory:              Local NTB Port:      Peer NTB Port:      Peer MMIO:
> +  ____________
> + | dma-mapped |-ntb_mw_set_trans(addr)  |
> + | memory     |        _v____________   |   ______________
> + | (addr)     |<======| MW xlat addr |<====| MW base addr |<== memory-mapped IO
> + |------------|       |--------------|  |  |--------------|
> +
> +So typical scenario of the first type memory window initialization looks:
> +1) allocate a memory region, 2) put translated address to NTB config,
> +3) somehow notify a peer device of performed initialization, 4) peer device
> +maps corresponding outbound memory window so to have access to the shared
> +memory region.
> +
> +The second type of interface, that implies the shared windows being
> +initialized by a peer device, is depicted on the figure:
> +
> +Outbound translation:
> + Memory:        Local NTB Port:    Peer NTB Port:      Peer MMIO:
> +  ____________                      ______________
> + | dma-mapped |                |   | MW base addr |<== memory-mapped IO
> + | memory     |                |   |--------------|
> + | (addr)     |<===================| MW xlat addr |<-ntb_peer_mw_set_trans(addr)
> + |------------|                |   |--------------|
> +
> +Typical scenario of the second type interface initialization would be:
> +1) allocate a memory region, 2) somehow deliver a translated address to a peer
> +device, 3) peer puts the translated address to NTB config, 4) peer device maps
> +outbound memory window so to have access to the shared memory region.
> +
> +As one can see the described scenarios can be combined in one portable
> +algorithm.
> + Local device:
> +  1) Allocate memory for a shared window
> +  2) Initialize memory window by translated address of the allocated region
> +     (it may fail if local memory window initialzation is unsupported)
> +  3) Send the translated address and memory window index to a peer device
> + Peer device:
> +  1) Initialize memory window with retrieved address of the allocated
> +     by another device memory region (it may fail if peer memory window
> +     initialization is unsupported)
> +  2) Map outbound memory window
> +
> +In accordance with this scenario, the NTB Memory Window API can be used as
> +follows:
> + Local device:
> +  1) ntb_mw_count(pidx) - retrieve number of memory ranges, which can
> +     be allocated for memory windows between local device and peer device
> +     of port with specified index.
> +  2) ntb_get_align(pidx, midx) - retrieve parameters restricting the
> +     shared memory region alignment and size. Then memory can be properly
> +     allocated.
> +  3) Allocate physically contiguous memory region in complience with
> +     restrictions retrieved in 2).
> +  4) ntb_mw_set_trans(pidx, midx) - try to set translation address of
> +     the memory window with specified index for the defined peer device
> +     (it may fail if local translated address setting is not supported)
> +  5) Send translated base address (usually together with memory window
> +     number) to the peer device using, for instance, scratchpad or message
> +     registers.
> + Peer device:
> +  1) ntb_peer_mw_set_trans(pidx, midx) - try to set received from other
> +     device (related to pidx) translated address for specified memory
> +     window. It may fail if retrieved address, for instance, exceeds
> +     maximum possible address or isn't properly aligned.
> +  2) ntb_peer_mw_get_addr(widx) - retrieve MMIO address to map the memory
> +     window so to have an access to the shared memory.
> +
> +Also it is worth to note, that method ntb_mw_count(pidx) should return the
> +same value as ntb_peer_mw_count() on the peer with port index - pidx.
> +
>  ### NTB Transport Client (ntb\_transport) and NTB Netdev (ntb\_netdev)
> 
>  The primary client for NTB is the Transport client, used in tandem with NTB
> --
> 2.6.6

^ permalink raw reply	[flat|nested] 62+ messages in thread
end of thread, other threads:[~2017-02-01 20:01 UTC | newest]

Thread overview: 62+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-11-29 17:15 [PATCH 00/22] NTB: Alter kernel API to support multi-port devices Serge Semin
2016-11-29 17:15 ` [PATCH 01/22] NTB: Move link state API being first in sources Serge Semin
2016-11-29 17:15 ` [PATCH 02/22] NTB: Add peer indexed ports NTB API Serge Semin
2016-11-30 18:40   ` kbuild test robot
2016-11-30 19:12   ` kbuild test robot
2016-11-30 20:04   ` kbuild test robot
2016-11-29 17:15 ` [PATCH 03/22] NTB: Alter NTB API to support both inbound and outbound MW based interfaces Serge Semin
2016-11-30 18:54   ` kbuild test robot
2016-11-30 19:46   ` kbuild test robot
2016-11-29 17:15 ` [PATCH 04/22] NTB: Add messaging NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 05/22] NTB: Alter Scratchpads NTB API to support multi-ports interface Serge Semin
2016-11-29 17:16 ` [PATCH 06/22] NTB: Slightly alter link state NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 07/22] NTB: Fix a few ntb.h issues Serge Semin
2016-11-29 17:16 ` [PATCH 08/22] NTB: Add T-Platforms copyrights to NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 09/22] NTB Intel: Move link-related methods being first in the driver Serge Semin
2016-11-29 17:16 ` [PATCH 10/22] NTB Intel: Add port-related NTB API callback methods Serge Semin
2016-12-07 22:56   ` Allen Hubbe
2016-11-29 17:16 ` [PATCH 11/22] NTB Intel: Alter MW interface to fit new NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 12/22] NTB Intel: Alter Scratchpads " Serge Semin
2016-11-29 17:16 ` [PATCH 13/22] NTB Intel: Add T-Platforms copyrights to Intel NTB driver Serge Semin
2016-11-29 17:16 ` [PATCH 14/22] NTB AMD: Move link-related methods being first in the driver Serge Semin
2016-11-29 17:16 ` [PATCH 15/22] NTB AMD: Add port-related NTB API callback methods Serge Semin
2016-11-29 17:16 ` [PATCH 16/22] NTB AMD: Alter MW interface to fit new NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 17/22] NTB AMD: Alter Scratchpads " Serge Semin
2016-11-29 17:16 ` [PATCH 18/22] NTB AMD: Add T-Platforms copyrights to AMD NTB driver Serge Semin
2016-11-29 17:16 ` [PATCH 19/22] NTB PingPong: Alter driver to work with two-ports NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 20/22] NTB Tool: " Serge Semin
2016-11-29 17:16 ` [PATCH 21/22] NTB Perf: " Serge Semin
2016-11-29 17:16 ` [PATCH 22/22] NTB Transport: " Serge Semin
2016-12-12 21:08 ` [PATCH v2 0/9] NTB: Alter kernel API to support multi-port devices Serge Semin
2016-12-12 21:08   ` [PATCH v2 1/9] NTB: Make link-state API being declared first Serge Semin
2016-12-12 21:08   ` [PATCH v2 2/9] NTB: Add indexed ports NTB API Serge Semin
2016-12-12 21:08   ` [PATCH v2 3/9] NTB: Alter link-state API to support multi-port devices Serge Semin
2016-12-12 21:08   ` [PATCH v2 4/9] NTB: Alter MW API to support multi-ports devices Serge Semin
2016-12-12 21:08   ` [PATCH v2 5/9] NTB: Alter Scratchpads " Serge Semin
2016-12-12 21:08   ` [PATCH v2 6/9] NTB: Add Messaging NTB API Serge Semin
2016-12-12 21:08   ` [PATCH v2 7/9] NTB: Add new Memory Windows API documentation Serge Semin
2016-12-12 21:08   ` [PATCH v2 8/9] NTB: Add PCIe Gen4 link speed Serge Semin
2016-12-12 21:08   ` [PATCH v2 9/9] NTB: Add ntb.h comments Serge Semin
2016-12-13 23:49   ` [PATCH v3 0/9] NTB: Alter kernel API to support multi-port devices Serge Semin
2016-12-13 23:49     ` [PATCH v3 1/9] NTB: Make link-state API being declared first Serge Semin
2016-12-14  7:07       ` Allen Hubbe
2016-12-13 23:49     ` [PATCH v3 2/9] NTB: Add indexed ports NTB API Serge Semin
2016-12-14  7:07       ` Allen Hubbe
2016-12-13 23:49     ` [PATCH v3 3/9] NTB: Alter link-state API to support multi-port devices Serge Semin
2016-12-13 23:49     ` [PATCH v3 4/9] NTB: Alter MW API to support multi-ports devices Serge Semin
2016-12-14  7:08       ` Allen Hubbe
2017-01-11  0:11       ` Serge Semin
2016-12-13 23:49     ` [PATCH v3 5/9] NTB: Alter Scratchpads " Serge Semin
2016-12-14  7:08       ` Allen Hubbe
2017-01-11  0:13       ` Serge Semin
2017-02-01 20:01         ` Jon Mason
2016-12-13 23:49     ` [PATCH v3 6/9] NTB: Add Messaging NTB API Serge Semin
2016-12-20  9:48       ` Serge Semin
2016-12-13 23:49     ` [PATCH v3 7/9] NTB: Add new Memory Windows API documentation Serge Semin
2016-12-13 23:49     ` [PATCH v3 8/9] NTB: Add PCIe Gen4 link speed Serge Semin
2016-12-13 23:49     ` [PATCH v3 9/9] NTB: Add ntb.h comments Serge Semin
2016-12-14  7:09       ` Allen Hubbe
2016-12-14  7:16       ` Serge Semin
2016-12-14  7:19         ` Allen Hubbe
2016-12-20  9:50         ` Serge Semin
  -- strict thread matches above, loose matches on Subject: below --
2016-12-13  4:14 [PATCH v2 7/9] NTB: Add new Memory Windows API documentation Allen Hubbe

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).