From: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
To: Viacheslav Dubeyko <slava@dubeyko.com>
Cc: <linux-cxl@vger.kernel.org>, <a.manzanares@samsung.com>,
<viacheslav.dubeyko@bytedance.com>
Subject: Re: CXL Fabric Manager (FM) architecture diagrams
Date: Tue, 7 Mar 2023 17:21:05 +0000 [thread overview]
Message-ID: <20230307172105.00006132@Huawei.com> (raw)
In-Reply-To: <20230306185913.1060918-1-slava@dubeyko.com>
On Mon, 6 Mar 2023 10:59:13 -0800
Viacheslav Dubeyko <slava@dubeyko.com> wrote:
> Hi Jonathan,
>
> You can find diagram online now:
>
> (1) Diagram 1: single host with Fabric Manager (FM)
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram1-cxl-fm-single-host-with-fm.pdf
Whilst accurate that you might only be able to control the switch, I would add one MLD to this
diagram so that tunneling can be discussed. It's minimal in sense of near minimal number of
components that might exist in a switched system
Thanks to asciiflow.com + some editing of the resutl - fingers crossed this works.
Obviously these can't be as rich as your nice diagrams so I've
left out what the connections are, but having them inline has
it's advantages as well.
I've played fast and loose with some of the terminology and not checked
it against the spec naming. I've also left it vague how an application
talks to the orchestrator. That was via agent in your terms I think
┌───────────────────┐ ┌───────────────┐ ┌────────────────┐
│ │ │ │ │ │
│ │ │ ┌───────┐ │ │ ┌────────────┐ │
│ │ │ │ ├───┼───┼─►FM Owned LD │ │
│ ┌───┐ │ │ │Switch │ │ │ └────────────┘ │
│ │FM ├───┼────┼───► CCI │ ├───┤ │
│ └───┘ │ │ │ │ │ │ MLD 1 │
│ │ │ └──────┬┘ │ └────────────────┘
│ ┌────┤ │ │ │
│ │RP0 ├────┤ │ │ ┌────────────────┐
│ └────┤ │ │ │ │ │
│ │ │ │ │ │ ┌────────────┐ │
│ ┌────┤ │ └────┼───┼─►FM Owned LD │ │
│ │RP1 ├────┤ ├───┤ └────────────┘ │
│ └────┤ │ │ │ │
│ Host A │ │ SWITCH │ │ MLD 2 │
└───────────────────┘ └───────────────┘ └────────────────┘
>
> (2) Diagram 2: single host with orchestrator
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram2-cxl-fm-single-host-with-orhestrator.pdf
Actually having an FM in a switch might happen, but there is no spec defined way of doing it.
From a software architecture point of view it's no different from another host talking to the
switch - just think of sticking a BMC down next to the switch chip.
Having the orchestrator in the host is also rather odd though it could in theory happen.
Typically the orchestrator is considered a 'cloud' level thing floating way above individual host.
Without any loss of generality I'd always have the orchestrator as something on it's own
machine chatting over a network to the Agents and FM-API accessed devices.
Something like
┌──────────────────────┐ ┌────────┐
│ │ │ │
│ │ │ │
│ Orchestrator ──────┼────► FM │
│ │ │ │
└───▲──────────────────┘ └─┬──────┘
│ │
┌─────┼─────────────┐ ┌──────┼────────┐ ┌────────────────┐
│ │ │ │ │ │ │ │
│ │ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
│ │ │ │ │ Switch ├───┼───┼─►FM Owned LD │ │
│ ┌───┴──┐ │ │ │ CCI │ │ │ └────────────┘ │
│ │ APP │ │ │ │ or MCTP│ ├───┤ │
│ │ │ │ │ │ CCI │ │ │ MLD 1 │
│ └──────┘ │ │ └───────┬┘ │ └────────────────┘
│ ┌────┤ │ │ │
│ │RP0 ├────┤ │ │ ┌────────────────┐
│ └────┤ │ │ │ │ │
│ │ │ │ │ │ ┌────────────┐ │
│ ┌────┤ │ └────┼───┼─►FM Owned LD │ │
│ │RP1 ├────┤ ├───┤ └────────────┘ │
│ └────┤ │ │ │ │
│ Host A │ │ SWITCH │ │ MLD 2 │
└───────────────────┘ └───────────────┘ └────────────────┘
>
> (3) Diagram3: Multi-headed device
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram3-cxl-fm-multi-headed-device.pdf
Looks good though there is a simpler single host version.
┌───────────────────┐ ┌─────────────────────┐
│ ┌───┐ │ │ ┌────────────┐ │
│ │FM ├───┼────┼─► Mailbox CCI│ │
│ └───┘ │ │ └────────────┘ │
│ │ │ ┌─────▼───────┐ │
│ ┌────┤ │ │ LD Pool CCI │ │
│ │RP0 ├────┤ └─────────────┘ │
│ └────┤ │ │
│ ┌────┤ │ │
│ │RP1 ├────┤ │
│ └────┤ │ │
│ Host A │ │ MHD 1 │
└───────────────────┘ └─────────────────────┘
I'd jump from single host to the multi host with external FM and
orchestator.
┌──────────────────────┐ ┌────────┐
│ │ │ │
│ │ │ │
┌─────► Orchestrator ──────┼────► FM │
│ │ │ │ │
│ └───▲──────────────────┘ └────┬───┘
│ │ │
│ ┌─────┼─────────────┐ ┌───────┼─────────────┐
│ │ │ │ │ ┌─────▼───────┐ │
│ │ │ │ │ │ Mailbox CCI │ │
│ │ ┌───┴──┐ ┌───┤ │ └─────-───────┘ │
│ │ │ APP │ │RP0├──────┤ │ │
│ │ │ │ └───┤ │ ┌─────▼───────┐ │
│ │ └──────┘ │ │ │ LD Pool CCI │ │
│ │ │ │ └─────────────┘ │
│ │ Host A │ │ │
│ └───────────────────┘ │ │
│ ┌───────────────────┐ │ │
└───┼─┬──────┐ ┌───┤ │ │
│ │ APP │ │RP0├──────┤ │
│ │ │ └───┤ │ │
│ └──────┘ │ │ │
│ Host B │ │ MHD 1 │
└───────────────────┘ └─────────────────────┘
>
> (4) Diagram 4: Multi-headed device + Orchestrator
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram4-cxl-fm-multi-headed-device-orchestrator.pdf
I'd put the orchestrator in it's own 'host' as above... I've drawn it with a mailbox cci but
could be a mctp CCI.
>
> (5) Diagram5: Multiple hosts with Fabric Manager (FM)
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram5-cxl-fm-multiple-hosts-with-fm.pdf
Another one where I'd separate the FM from the switch. It may be near the switch but
it's talking fm-api to the switch and that's an interface that is well defined.
┌──────────────────────┐ ┌────────┐
┌─────► Orchestrator ──────┼────► FM │
│ └───▲──────────────────┘ └────┬───┘
│ ┌─────┼─────────────┐ │
│ │ │ │ ┌──────┼────────┐ ┌────────────────┐
│ │ │ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
│ │ ┌───┴──┐ ┌───┤ │ │ Switch ├───┼───┼─►FM Owned LD │ │
│ │ │ APP │ │RP0├───────┤ │ CCI │ │ │ └────────────┘ │
│ │ │ │ └───┤ │ │ or MCTP│ ├───┤ │
│ │ └──────┘ │ │ │ CCI │ │ │ MLD 1 │
│ │ │ │ └───────┬┘ │ └────────────────┘
│ │ Host A │ │ │ │
│ └───────────────────┘ │ │ │ ┌────────────────┐
│ ┌───────────────────┐ │ │ │ │ ┌────────────┐ │
│ │ │ │ └────┼───┼─►FM Owned LD │ │
│ │ │ │ ├───┤ └────────────┘ │
│ │ │ │ │ │ │
└───┼─┬──────┐ ┌───┤ │ SWITCH │ │ MLD 2 │
│ │ APP │ │RP0├───────┤ │ └────────────────┘
│ │ │ └───┤ │ │
│ └──────┘ │ └───────────────┘
│ Host B │
└───────────────────┘
>
> (6) Diagram 6: Multiple hosts with orhestrator
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram6-cxl-fm-multiple-hosts-with-orhestrator.pdf
I'm not sure there is a need to separate the case where there
is an orchestrator in the loop from when there isn't. Hence I just threw one on the diagram above.
>
> (7) Diagram 7: Distributed Fabric Manager (FM)
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram7-cxl-fm-distributed-fm.pdf
>
Looks like a set of FMs, not what I'd think of as a distributed FM.
Only makes sense to me if more like this...
┌──────────────────────┐
┌───► Orchestrator ──────┼─────────────────────────────────────────┐
│ │ ▲ │ │
│ └───┬──────────┬───────┘ │
│ │ │ │
│ │ │ ┌────────┐ │
│ │ └────────────► FM1 │ │
│ │ └────┬───┘ │
│ ┌─────┼─────────────┐ │ │
│ │ │ │ ┌──────┼────────┐ ┌────────────────┐ │
│ │ │ ┌───┤ │ │ │ │ │ │
│ │ │ │RP0├───────┤ ┌───▼────┐ │ │ ┌────────────┐ │ │
│ │ ┌───┴──┐ └───┤ │ │ Switch ├───┼───┼─►FM Owned LD │ │ │
│ │ │ APP │ │ │ │ CCI │ │ │ └────────────┘ │ │
│ │ │ │ ┌───┤ │ │ or MCTP│ ├───┤ │ │
│ │ └──────┘ │RP1├─────┐ │ │ CCI │ │ │ MLD 1 │ │
│ │ └───┤ │ │ └───────┬┘ │ └────────────────┘ │
│ │ Host A │ │ │ │ │ │
│ └───────────────────┘ │ │ │ │ ┌────────────────┐ │
│ │ │ │ │ │ │ │
│ ┌───────────────────┐ │ │ │ │ │ ┌────────────┐ │ │
│ │ │ │ │ └────┼───┼─►FM Owned LD │ │ │
│ │ ┌───┤ │ │ ├───┤ └────────────┘ │ │
│ │ │RP0├─────┼─┤ │ │ │ │
│ │ ┌──────┐ └───┤ │ │ SWITCH 1 │ │ MLD 2 │ │
└─┼─┤ APP │ │ │ │ │ └────────────────┘ │
│ │ │ ┌───┤ │ │ │ │
│ └──────┘ │RP1├──┐ │ └───────────────┘ │
│ └───┤ │ │ │
│ Host B │ │ │ │
└───────────────────┘ │ │ │
│ │ ┌────────┐ │
│ │ │ FM2 ◄───────────────────────────┘
│ │ └────┬───┘
│ │ ┌──────┼────────┐ ┌────────────────┐
│ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
│ │ │ │ Switch ├───┼───┼─►FM Owned LD │ │
│ └─┤ │ CCI │ │ │ └────────────┘ │
│ │ │ or MCTP│ ├───┤ │
│ │ │ CCI │ │ │ MLD 3 │
│ │ └───────┬┘ │ └────────────────┘
│ │ │ │ ┌────────────────┐
│ │ │ │ │ ┌────────────┐ │
│ │ └────┼───┼─►FM Owned LD │ │
│ │ ├───┤ └────────────┘ │
│ │ │ │ │
│ │ SWITCH 2 │ │ MLD 4 │
└────┤ │ └────────────────┘
└───────────────┘
> (8) Diagram 8: Layered Fabric Manager (FM) and separate orchestrator
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram8-cxl-fm-layered-fm-and-separate-orchestrator.pdf
I don't follow the layered part on this diagram. My interpretation would
be something like.
┌──────────────────────┐ ┌────────┐
┌─────► Orchestrator ──────┼────► TOP FM ├──────────────────────────────┐
│ │ ▲ │ │ │ │
│ └───┬──────────────────┘ └────┬───┘ │
│ │ │ │
│ │ ┌────┴───┐ │
│ │ │ BOTTOM │ │
│ │ │ FM1 │ │
│ │ └────┬───┘ │
│ ┌─────┼─────────────┐ │ │
│ │ │ │ ┌──────┼────────┐ ┌────────────────┐ │
│ │ │ ┌───┤ │ │ │ │ │ │
│ │ │ │RP0├───────┤ ┌───▼────┐ │ │ ┌────────────┐ │ │
│ │ ┌───┴──┐ └───┤ │ │ Switch ├───┼───┼─►FM Owned LD │ │ │
│ │ │ APP │ │ │ │ CCI │ │ │ └────────────┘ │ │
│ │ │ │ ┌───┤ │ │ or MCTP│ ├───┤ │ │
│ │ └──────┘ │RP1├─────┐ │ │ CCI │ │ │ MLD 1 │ │
│ │ └───┤ │ │ └───────┬┘ │ └────────────────┘ │
│ │ Host A │ │ │ │ │ │
│ └───────────────────┘ │ │ │ │ ┌────────────────┐ │
│ │ │ │ │ │ │ │
│ │ │ │ │ │ ┌────────────┐ │ │
│ ┌───────────────────┐ │ │ └────┼───┼─►FM Owned LD │ │ │
│ │ ┌───┤ │ │ ├───┤ └────────────┘ │ │
│ │ │RP0├─────┼─┤ │ │ │ │
│ │ ┌──────┐ └───┤ │ │ SWITCH 1 │ │ MLD 2 │ │
└───┼─┤ APP │ │ │ │ │ └────────────────┘ │
│ │ │ ┌───┤ │ │ │ │
│ └──────┘ │RP1├──┐ │ └───────────────┘ │
│ └───┤ │ │ │
│ Host B │ │ │ │
└───────────────────┘ │ │ │
│ │ ┌────────┐ │
│ │ │ BOTTOM │ │
│ │ │ FM2 ◄──────────────────────────────┘
│ │ └────┬───┘
│ │ ┌──────┼────────┐ ┌────────────────┐
│ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
│ │ │ │ Switch ├───┼───┼─►FM Owned LD │ │
│ └─┤ │ CCI │ │ │ └────────────┘ │
│ │ │ or MCTP│ ├───┤ │
│ │ │ CCI │ │ │ MLD 3 │
│ │ └───────┬┘ │ └────────────────┘
│ │ │ │ ┌────────────────┐
│ │ │ │ │ ┌────────────┐ │
│ │ └────┼───┼─►FM Owned LD │ │
│ │ ├───┤ └────────────┘ │
│ │ SWITCH 2 │ │ MLD 4 │
└────┤ │ └────────────────┘
└───────────────┘
>
> (9) Diagram 9: BMC based layered Fabric Manager (FM)
> https://github.com/computexpresslink/cxl-fm-architecture/blob/main/diagram9-cxl-fm-bmc-based-layered-fm.pdf
I don't follow what the BMC is in this diagram.
The BMC is just a (cheap) host that happens to have some elements of the overall management
infrastructure on it. Let it be any of the FMs floating around on their own in the
diagrams above. The diagram immediately above might be built with 3 BMCs or
as a single BMC like this...
┌──────────────────────┐
┌───► Orchestrator │
│ └───▲──────────┬───────┘ ┌────────┐
│ │ │ │ ┌────┐ │
│ │ └────────────►─┤FM │ ├───────────────────────────┐
│ │ │ └──┬─┘ │ │
│ │ │BMC │ │ │
│ │ └────┼───┘ │
│ ┌─────┼─────────────┐ │ │
│ │ │ │ ┌──────┼────────┐ ┌────────────────┐ │
│ │ │ ┌───┤ │ │ │ │ │ │
│ │ │ │RP0├───────┤ ┌───▼────┐ │ │ ┌────────────┐ │ │
│ │ ┌───┴──┐ └───┤ │ │ Switch ├───┼───┼─►FM Owned LD │ │ │
│ │ │ APP │ │ │ │ CCI │ │ │ └────────────┘ │ │
│ │ │ │ ┌───┤ │ │ or MCTP│ ├───┤ │ │
│ │ └──────┘ │RP1├─────┐ │ │ CCI │ │ │ MLD 1 │ │
│ │ └───┤ │ │ └───────┬┘ │ └────────────────┘ │
│ │ Host A │ │ │ │ │ │
│ └───────────────────┘ │ │ │ │ ┌────────────────┐ │
│ │ │ │ │ │ │ │
│ ┌───────────────────┐ │ │ │ │ │ ┌────────────┐ │ │
│ │ │ │ │ └────┼───┼─►FM Owned LD │ │ │
│ │ ┌───┤ │ │ ├───┤ └────────────┘ │ │
│ │ │RP0├─────┼─┤ │ │ │ │
│ │ ┌──────┐ └───┤ │ │ SWITCH 1 │ │ MLD 2 │ │
└─┼─┤ APP │ │ │ │ │ └────────────────┘ │
│ │ │ ┌───┤ │ │ │ │
│ └──────┘ │RP1├──┐ │ └───────────────┘ │
│ └───┤ │ │ │
│ Host B │ │ │ │
└───────────────────┘ │ │ │
│ │ ┌───────────────────────────────┘
│ │ ┌──────┼────────┐ ┌────────────────┐
│ │ │ ┌───▼────┐ │ │ ┌────────────┐ │
│ │ │ │ Switch ├───┼───┼─►FM Owned LD │ │
│ └─┤ │ CCI │ │ │ └────────────┘ │
│ │ │ or MCTP│ ├───┤ │
│ │ │ CCI │ │ │ MLD 3 │
│ │ └───────┬┘ │ └────────────────┘
│ │ │ │ ┌────────────────┐
│ │ │ │ │ ┌────────────┐ │
│ │ └────┼───┼─►FM Owned LD │ │
│ │ ├───┤ └────────────┘ │
│ │ SWITCH 2 │ │ MLD 4 │
└────┤ │ └────────────────┘
└───────────────┘
>
> So, have I missed something?
> Should we correct something on diagrams?
> Does it look good?
Thare are far too many things we should perhaps show on these diagrams, but
I suspect it will mostly fall out of any layered design.
We could draw one incredibly complex diagram that does everything :)
*crossed fingers the ascii art fun above looks ok*
Thanks for getting this started btw! I was completely failing to start
whereas once there was a proposal it became easier to have a go!
Jonathan
>
> Thanks,
> Slava.
next prev parent reply other threads:[~2023-03-07 17:26 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-03-06 18:59 CXL Fabric Manager (FM) architecture diagrams Viacheslav Dubeyko
2023-03-07 17:21 ` Jonathan Cameron [this message]
2023-03-09 22:28 ` [External] " Viacheslav A.Dubeyko
[not found] ` <20230310120126.000057b9@Huawei.com>
2023-03-16 20:43 ` Viacheslav A.Dubeyko
[not found] ` <20230406204227.GA5971@bgt-140510-bm01>
2023-04-07 18:18 ` Viacheslav A.Dubeyko
[not found] ` <90aea04c-dab2-4b7e-bfc6-09a1240793a9@nmtadam.samsung>
2023-04-07 23:24 ` Viacheslav A.Dubeyko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230307172105.00006132@Huawei.com \
--to=jonathan.cameron@huawei.com \
--cc=a.manzanares@samsung.com \
--cc=linux-cxl@vger.kernel.org \
--cc=slava@dubeyko.com \
--cc=viacheslav.dubeyko@bytedance.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox