RE: Is the Scratchpad Implementation Using a LUT Standard?

["Allen Hubbe" <Allen.Hubbe@dell.com>]

From: Doug Meyer
> Gents,
> 
> As I continue to gain understanding about the NTB code, I am wondering about the use and requirements
> surrounding scratchpads, at least as I see things in the Switchtec code.
> 
> In there, I see that a LUT (LUT0) is used, the size of the LUTs apparently being hard-coded to 64 KiB,
> for a shared memory window (struct shared_mw) which contains an array of 128 u32 for the scratchpad.
> Also, it appears that this is what is used both to determine link status and to pass memory
> address/size information between hosts (ports, peers). I apologize if I have that wrong. Please
> correct me.

The struct shared_mw is unique to the Switchtec driver, and your interpretation of its mechanism matches my understanding.

Other drivers determine link state from the hardware, and only expose scratchpads if they are implemented in hardware.

> The goal here is just to gain understanding... learn about required APIs vs philosophical decisions vs
> convenience, etc.
> 
> My questions are whether this is a fixture of the NTB architecture, or if this is a convenience to
> support something else (the latter being a requirement)?
> 
> In particular,
> 
> But most importantly, I'm wondering about struct shared_mw. Could the Switchtec message registers have
> been used?

About a year ago when Serge joined the team, we spent a while trying to unify the message and scratchpad api.  At the same time, there is a preference to keep ntb.h very light and expose the hardware functionality as directly as possible.  We decided to split the apis for scratchpads and message registers.  For hardware that supports message registers, it should expose those via the message api.

What this currently implies is that the next layer up driver needs to work with either scratchpads or message registers.  If a driver only works with spads, then it is not portable.  I would like there to be some library code added to the common ntb bus driver to help with that.  Serge is currently making changes to the ntb_transport driver to support multi-port and message registers on IDT.  It may only work for the transport driver at first, but I have some hope that it could be transformed into library code.

> What do people think about how this technique scales when there are more than two peers? Obviously

That limitation was stated upfront with that driver submission.  The Switchtec driver only works with two nodes for now.

> LUTs are a precious resource, and a LUT per peer shared_mw is expensive. A LUT broken up into many
> shared_mw is a possibility, though there is always risk of trashing stuff.
> Also, if the LUTs need to be much larger (for application use), then a large chunk of the BAR space
> could be used for a relatively small structure.
> I'd love to hear anyone's thoughts on this.
> 
> As an aside, I'm curious why the LUT size is 64 KiB? Was this just a nice number as a starting point?
> 
> Thanks again, folks.
> 
> Blessings,
> Doug