Re: Is the Scratchpad Implementation Using a LUT Standard?

[Logan Gunthorpe <lsgunthorpe@gmail.com>]

On 11/10/17 12:23 PM, D Meyer wrote:
> Regarding Switchtec, the 96xG3 part has 512 LUTs per Stack and a stack
> can have up to eight NTBs. So for that chip, if I'm starting to grasp
> this stuff, a machine with 48 NTBs would mean that on a single Stack,
> 8 (hosts) x 47 (peers) = 376 LUTs would be used up for each of the 8
> to have a shared_mw in its own LUT.
Yes, there are a bunch of annoying restrictions like that, but your 
example looks correct.
There are lots of LUTs to play around with. The bigger restriction is 
the direct windows, of
which you only have 2 per port. Creating a network in ntb_transport (et 
al) to communicate across
48 partitions is going to be a very hard problem to solve.
> Also, the current hard-coded LUT size is 64 KiB, but that can't remain
> that way to have flexibility. If the LUTs were, say 16 MiB, then the
> shred_mw LUTs use up a massive part of the BAR space.
Yes, all LUTs must be the same size. Plus the LUT space comes before the 
direct
window space in the BAR. So the alignment (and therefore maximum size) 
of the direct window space
depends on the size and the number of LUTs. I believe I chose 32 64k 
luts so that the
direct window aligns to 2MB. I originally had the size set to 4k, but 
this limited the alignment
of the direct window.

So the tradeoffs are: if you increase it you waste memory for LUTs that 
don't need the extra space,
and if you decrease it you limit the size of the direct window.
> Oh... A quick side question: For swithtec, I see that the number of
> available LUTs for a BAR is read out of the chip and then rounded down
> to a power of two... I'm curious why it's rounded down?
This has to do with the alignment I mentioned above. If the number of 
LUTs is not a power of two,
the direct window won't be nicely aligned and you get some annoying 
restrictions on it's size.

Logan