[LSF/MM/BPF Topic][LSF/MM/BPF Attend] iscsi issue of scale with MNoT

[LSF/MM/BPF Topic][LSF/MM/BPF Attend] iscsi issue of scale with MNoT

[lduncan]

[RESEND -- apologies if you see this more than once]

The iSCSI protocol continues to be used in Linux, but some of the
users push the system past its normal limits. And using multipath just
exacerbates that problem (usually doubling the number of sessions).

I'd like to gather some numbers for open-iscsi (the standard Linux
iSCSI initiator) and the kernel target code (i.e. LIO/targetcli) on
what happens when there are MNoT -- massive numbers of targets.

"Massive" in my case, will be relative, since I don't have access to
a supercomputer, but I believe it will not be too hard to start
pushing the system too far. For example, a recent user problem found
that even at 2000 sessions using multipath, the system takes about 80
seconds to switch paths. Each switch takes 80ms (and they are
currently serialized), but when you multiply that by 1000 it adds up.

For the initiator, I've long suspected some parts of the code were not
designed for scale, so this might give me a chance to find and
possibly address some of these issues.

--
Lee Duncan

Re: [LSF/MM/BPF Topic][LSF/MM/BPF Attend] iscsi issue of scale with MNoT

[Khazhy Kumykov]

[-- Attachment #1: Type: text/plain, Size: 1437 bytes --]

On Thu, Feb 17, 2022 at 10:19 AM <lduncan@suse.com> wrote:
>
> [RESEND -- apologies if you see this more than once]
>
> The iSCSI protocol continues to be used in Linux, but some of the
> users push the system past its normal limits. And using multipath just
> exacerbates that problem (usually doubling the number of sessions).
>
> I'd like to gather some numbers for open-iscsi (the standard Linux
> iSCSI initiator) and the kernel target code (i.e. LIO/targetcli) on
> what happens when there are MNoT -- massive numbers of targets.
>
> "Massive" in my case, will be relative, since I don't have access to
> a supercomputer, but I believe it will not be too hard to start
> pushing the system too far. For example, a recent user problem found
> that even at 2000 sessions using multipath, the system takes about 80
> seconds to switch paths. Each switch takes 80ms (and they are
> currently serialized), but when you multiply that by 1000 it adds up.
>
> For the initiator, I've long suspected some parts of the code were not
> designed for scale, so this might give me a chance to find and
> possibly address some of these issues.

There are some linear lookups (sess_list, conn_list) in the iSCSI
netlink which may be low hanging fruit, and do show up in heatmaps
when creating/destroying sessions/connections in the presence of 10k+
sessions in a machine. (Though this doesn't manifest 80s+ stalls,
thankfully)
>
> --
> Lee Duncan

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3996 bytes --]