Re: Linux SCTP associations failure handlig

Re: Linux SCTP associations failure handlig

[Marcelo Ricardo Leitner]

Hi oaitamrane,

On Wed, Jul 06, 2022 at 01:19:20PM -0000, oaitamrane@users.sourceforge.net wrote:
> 
> 
> Dear Marcelo
> 
> I would like to know if there is a way after association failure (SCTP_COMM_LOST event treceived) to retrieve UNSENT to peers and UNACKNOWLEDGED by peers Data Chunks for retransmission over an alternate association?
> I am using RHEL8.4 and sockets are set in NON BLOCKING mode.
> I have set the sctp_event_subscribe structure as follows:
> 
>   memset(&sctp_events, 0, sizeof(sctp_events));
> 
>   sctp_events.sctp_data_io_event = 1;
>   sctp_events.sctp_association_event = 1;
>   sctp_events.sctp_address_event =  1;
>   sctp_events.sctp_peer_error_event = 1;
>   sctp_events.sctp_adaptation_layer_event = 1;
>   sctp_events.sctp_shutdown_event = 1;
>   sctp_events.sctp_partial_delivery_event = 1;
>   sctp_events.sctp_send_failure_event_event = 1;
> 
>   ret = setsockopt(sock_fd, IPPROTO_SCTP, SCTP_EVENTS,
> &sctp_events, sizeof(sctp_events));

My understanding is that you should be getting it via
sctp_send_failure_event_event above, detailed in rfc6458#section-6.1.11.

When SCTP tears down an association, it marks the chunks in outqueue
as failed in __sctp_outq_teardown(), which then sctp_datamsg_destroy()
ends up picking up and sending notifications to the application.

I didn't check this on RHEL 8.4, though, but I believe it has support
for this RFC as well. If you have the event as above, it should be
there.

> 
> Can you please tell me if you a member of the dev team who can help me.

+Cc linux-sctp here to have a broader reach. We don't really use
sourceforge anymore. :-)

Cheers,
Marcelo

> 
> Regards
> Omar AIT AMRANE
> 
> - - - - - - - - - - - - - - - - - - - - -
> 
> This message was sent to you via the SourceForge web mail form.
> You may reply to this message directly, or at https://sourceforge.net/u/oaitamrane/profile/send_message

Re: Linux SCTP associations failure handlig

[o.evistel]

Hi Marcelo

Thank you for your help.

Regards
Omar AIT AMRANE
==============================================================

----- Mail original -----
De: "Marcelo Ricardo Leitner" <marcelo.leitner@gmail.com>
À: "o" <o.evistel@free.fr>
Cc: oaitamrane@users.sourceforge.net, linux-sctp@vger.kernel.org
Envoyé: Jeudi 7 Juillet 2022 15:39:26
Objet: Re: Linux SCTP associations failure handlig

Hi oaitamrane,

On Wed, Jul 06, 2022 at 01:19:20PM -0000, oaitamrane@users.sourceforge.net wrote:
> 
> 
> Dear Marcelo
> 
> I would like to know if there is a way after association failure (SCTP_COMM_LOST event treceived) to retrieve UNSENT to peers and UNACKNOWLEDGED by peers Data Chunks for retransmission over an alternate association?
> I am using RHEL8.4 and sockets are set in NON BLOCKING mode.
> I have set the sctp_event_subscribe structure as follows:
> 
>   memset(&sctp_events, 0, sizeof(sctp_events));
> 
>   sctp_events.sctp_data_io_event = 1;
>   sctp_events.sctp_association_event = 1;
>   sctp_events.sctp_address_event =  1;
>   sctp_events.sctp_peer_error_event = 1;
>   sctp_events.sctp_adaptation_layer_event = 1;
>   sctp_events.sctp_shutdown_event = 1;
>   sctp_events.sctp_partial_delivery_event = 1;
>   sctp_events.sctp_send_failure_event_event = 1;
> 
>   ret = setsockopt(sock_fd, IPPROTO_SCTP, SCTP_EVENTS,
> &sctp_events, sizeof(sctp_events));

My understanding is that you should be getting it via
sctp_send_failure_event_event above, detailed in rfc6458#section-6.1.11.

When SCTP tears down an association, it marks the chunks in outqueue
as failed in __sctp_outq_teardown(), which then sctp_datamsg_destroy()
ends up picking up and sending notifications to the application.

I didn't check this on RHEL 8.4, though, but I believe it has support
for this RFC as well. If you have the event as above, it should be
there.

> 
> Can you please tell me if you a member of the dev team who can help me.

+Cc linux-sctp here to have a broader reach. We don't really use
sourceforge anymore. :-)

Cheers,
Marcelo

> 
> Regards
> Omar AIT AMRANE
> 
> - - - - - - - - - - - - - - - - - - - - -
> 
> This message was sent to you via the SourceForge web mail form.
> You may reply to this message directly, or at https://sourceforge.net/u/oaitamrane/profile/send_message

Linux SCTP performance question

[o.evistel]

Dear Marcelo

I am using linux-sctp as transport for SIGTRAN M3UA on RHEL 8.4 and I am using sctp_sendmsg() and sctp_recvmsg() to send/receive.
I would like to know if the use of sctp_sendv() and sctp_recvv() enhances performances ?

Regards
Omar AIT AMRANE

Re: Linux SCTP performance question

[Marcelo Ricardo Leitner]

Hi Omar,

Both will end up sending the data with sendmsg(), but sctp_sendv()
will allow more flexibility. Unless you save some syscalls because of
it, performace wise it should be very similar.

Regards,
Marcelo

On Wed, Jul 13, 2022 at 02:58:40PM +0200, o.evistel@free.fr wrote:
> Dear Marcelo
> 
> I am using linux-sctp as transport for SIGTRAN M3UA on RHEL 8.4 and I am using sctp_sendmsg() and sctp_recvmsg() to send/receive.
> I would like to know if the use of sctp_sendv() and sctp_recvv() enhances performances ?
> 
> Regards
> Omar AIT AMRANE
> 

Re: Linux SCTP performance question

[o.evistel]

Thank you Marcelo for your help.

Omar AIT AMRANE
===================================



----- Mail original -----
De: "Marcelo Ricardo Leitner" <marcelo.leitner@gmail.com>
À: "o" <o.evistel@free.fr>
Cc: "linux-sctp" <linux-sctp@vger.kernel.org>
Envoyé: Vendredi 15 Juillet 2022 20:48:58
Objet: Re: Linux SCTP performance question

Hi Omar,

Both will end up sending the data with sendmsg(), but sctp_sendv()
will allow more flexibility. Unless you save some syscalls because of
it, performace wise it should be very similar.

Regards,
Marcelo

On Wed, Jul 13, 2022 at 02:58:40PM +0200, o.evistel@free.fr wrote:
> Dear Marcelo
> 
> I am using linux-sctp as transport for SIGTRAN M3UA on RHEL 8.4 and I am using sctp_sendmsg() and sctp_recvmsg() to send/receive.
> I would like to know if the use of sctp_sendv() and sctp_recvv() enhances performances ?
> 
> Regards
> Omar AIT AMRANE
>

RE: Linux SCTP performance question

[David Laight]

From: o.evistel@free.fr
> Sent: 13 July 2022 13:59
> 
> I am using linux-sctp as transport for SIGTRAN M3UA on RHEL 8.4 and I am
> using sctp_sendmsg() and sctp_recvmsg() to send/receive.
> I would like to know if the use of sctp_sendv() and sctp_recvv() enhances performances ?

Every copy to/from a user buffer has a small (but measurable) cost.
So anything with an iov[] array is a bit slower than an equivalent
call that only has a single buffer.
This is measurable when comparing sendmsg() and sendto() on (say)
a UDP socket.
OTOH it is all probably noise unless you are trying to send/receive
'silly numbers' of messages.

Reducing the number of system calls may help.
But the cost of recvmmsg() checking for a second message is
significantly greater than using epoll().
So unless you actually expect lots of messages it probably
isn't worth using.

Assuming you have disabled Nagle, then the biggest performance
gain will be from setting MSG_MORE when you know you have another
message to send.
Without that pretty much every M3UA message ends up in its own
ethernet packet.

You don't mention why you think you have a performance issue in SCTP.
I think we got 40000 reflects/sec from a tcap application over M3UA
and SCTP last time I measured out stack.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: Linux SCTP performance question

[o.evistel]

Hi David

I try to perform a performance test with a configuration consisting of a M3UA-ASP VM (client) served by 4 M3UA-SGP VMs (servers).
All the VMs are RHEL8.4.
The TCAP traffic is initiated by the ASP (10K DATA-Chunk/sec) and evenly distributed over the 4 SGPs which reflect the received traffic to the ASP.
Within a random interval (15 min to several hours) the ASP experiences an error when calling sctp_sendv() with ERRNO="Resource Temporarily Unavailable".
As I don't know what causes theese errors I am wondering if this is not a performance issue.

Referring to your performance test:
- Does 40K reflects/sec mean 40K Tx DATA-Chunk/sec and 40K Rx DATA-Chunk/sec ?
- Did you apply specific kernel parameters tuning (rmem, wmem, ...) ?
- Can you share the test program you were using ?

Thank you in advance for your help and support.

Regards
Omar AIT AMRANE
========================================================================================


----- Mail original -----
De: "David Laight" <David.Laight@ACULAB.COM>
À: "o" <o.evistel@free.fr>, "Marcelo Ricardo Leitner" <marcelo.leitner@gmail.com>
Cc: "linux-sctp" <linux-sctp@vger.kernel.org>
Envoyé: Samedi 16 Juillet 2022 14:13:31
Objet: RE: Linux SCTP performance question

From: o.evistel@free.fr
> Sent: 13 July 2022 13:59
> 
> I am using linux-sctp as transport for SIGTRAN M3UA on RHEL 8.4 and I am
> using sctp_sendmsg() and sctp_recvmsg() to send/receive.
> I would like to know if the use of sctp_sendv() and sctp_recvv() enhances performances ?

Every copy to/from a user buffer has a small (but measurable) cost.
So anything with an iov[] array is a bit slower than an equivalent
call that only has a single buffer.
This is measurable when comparing sendmsg() and sendto() on (say)
a UDP socket.
OTOH it is all probably noise unless you are trying to send/receive
'silly numbers' of messages.

Reducing the number of system calls may help.
But the cost of recvmmsg() checking for a second message is
significantly greater than using epoll().
So unless you actually expect lots of messages it probably
isn't worth using.

Assuming you have disabled Nagle, then the biggest performance
gain will be from setting MSG_MORE when you know you have another
message to send.
Without that pretty much every M3UA message ends up in its own
ethernet packet.

You don't mention why you think you have a performance issue in SCTP.
I think we got 40000 reflects/sec from a tcap application over M3UA
and SCTP last time I measured out stack.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

RE: Linux SCTP performance question

[David Laight]

From: o.evistel@free.fr
> Sent: 18 July 2022 15:10
> Hi David
> 
> I try to perform a performance test with a configuration consisting of a M3UA-ASP VM (client) served
> by 4 M3UA-SGP VMs (servers).
> All the VMs are RHEL8.4.
> The TCAP traffic is initiated by the ASP (10K DATA-Chunk/sec) and evenly distributed over the 4 SGPs
> which reflect the received traffic to the ASP.
> Within a random interval (15 min to several hours) the ASP experiences an error when calling
> sctp_sendv() with ERRNO="Resource Temporarily Unavailable".

That's looks like a non-blocking send hitting the socket write buffer limit.
Almost certainly very transient.
Possibly cause by dropped packets or the receiving being overloaded.

> As I don't know what causes theese errors I am wondering if this is not a performance issue.
> 
> Referring to your performance test:
> - Does 40K reflects/sec mean 40K Tx DATA-Chunk/sec and 40K Rx DATA-Chunk/sec ?

Yes.
It will be a 'double-reflect' test (both ends reflect) with a smallish
number of packets in the loop - enough to get maximum throughput
without packets being discarded.

> - Did you apply specific kernel parameters tuning (rmem, wmem, ...) ?

No.

> - Can you share the test program you were using ?

It won't help you.
It uses our proprietary protocol over TCP to pass TCAP messages into
a driver (mostly SCCP, ISUP and MTP3) which uses kernel sockets to do
the actual send/recv.

> 
> Thank you in advance for your help and support.
> 
> Regards
> Omar AIT AMRANE
> ========================================================================================
> 
> 
> ----- Mail original -----
> De: "David Laight" <David.Laight@ACULAB.COM>
> À: "o" <o.evistel@free.fr>, "Marcelo Ricardo Leitner" <marcelo.leitner@gmail.com>
> Cc: "linux-sctp" <linux-sctp@vger.kernel.org>
> Envoyé: Samedi 16 Juillet 2022 14:13:31
> Objet: RE: Linux SCTP performance question
> 
> From: o.evistel@free.fr
> > Sent: 13 July 2022 13:59
> >
> > I am using linux-sctp as transport for SIGTRAN M3UA on RHEL 8.4 and I am
> > using sctp_sendmsg() and sctp_recvmsg() to send/receive.
> > I would like to know if the use of sctp_sendv() and sctp_recvv() enhances performances ?
> 
> Every copy to/from a user buffer has a small (but measurable) cost.
> So anything with an iov[] array is a bit slower than an equivalent
> call that only has a single buffer.
> This is measurable when comparing sendmsg() and sendto() on (say)
> a UDP socket.
> OTOH it is all probably noise unless you are trying to send/receive
> 'silly numbers' of messages.
> 
> Reducing the number of system calls may help.
> But the cost of recvmmsg() checking for a second message is
> significantly greater than using epoll().
> So unless you actually expect lots of messages it probably
> isn't worth using.
> 
> Assuming you have disabled Nagle, then the biggest performance
> gain will be from setting MSG_MORE when you know you have another
> message to send.
> Without that pretty much every M3UA message ends up in its own
> ethernet packet.
> 
> You don't mention why you think you have a performance issue in SCTP.
> I think we got 40000 reflects/sec from a tcap application over M3UA
> and SCTP last time I measured out stack.
> 
> 	David
> 
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
> Registration No: 1397386 (Wales)

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

RE: Linux SCTP performance question

[David Laight]

From: o.evistel@free.fr
> Sent: 18 July 2022 15:10
> 
> I try to perform a performance test with a configuration consisting
> of a M3UA-ASP VM (client) served by 4 M3UA-SGP VMs (servers).
> All the VMs are RHEL8.4.

Find some real systems....

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Linux SCTP multihoming question

[o.evistel]

Dear Marcelo

I am using linux-sctp as transport for SIGTRAN M3UA on RHEL 8.4 with multihoming (sctp_bindx(), sctp_connectx() API functions).
I would like to know, after association setup, if it is possible to instruct SCTP to use a specific local address from the list of bound addresses to reach the peer.

Regards
Omar AIT AMRANE

Re: Linux SCTP multihoming question

[Andreas Fink]

yes this is possible. My eSTP does this exactly this way. for M3UA and M2PA You simply use sctp_bindx to bind to exact local address you need.
You can add or remove local IP's and alter the association. But I'm not sure if you can force a specific path to be used as primary. SCTP basically can use any and switch over to the other.

If you think of something like hot/standby setup, you might want to do this on M3UA level which means having two SCTP associations (two ASPs in one AS)



> On 9 Feb 2024, at 11:05, o.evistel@free.fr wrote:
> 
> 
> Dear Marcelo
> 
> I am using linux-sctp as transport for SIGTRAN M3UA on RHEL 8.4 with multihoming (sctp_bindx(), sctp_connectx() API functions).
> I would like to know, after association setup, if it is possible to instruct SCTP to use a specific local address from the list of bound addresses to reach the peer.
> 
> Regards
> Omar AIT AMRANE
> 

Re: Linux SCTP multihoming question

[o.evistel]

Hi Andreas,

My explanation was not clear enough.

I would like to know if it is possible to force sctp to use alernate address instead of primary address while both of them are available.

Regards
Omar 
=========================================================================================

----- Mail original -----
De: "Andreas Fink" <afink@list.fink.org>
À: "OMAR AIT AMRANE" <o.evistel@free.fr>
Cc: "Marcelo Ricardo Leitner" <marcelo.leitner@gmail.com>, "linux-sctp" <linux-sctp@vger.kernel.org>
Envoyé: Vendredi 9 Février 2024 11:10:22
Objet: Re: Linux SCTP multihoming question

yes this is possible. My eSTP does this exactly this way. for M3UA and M2PA You simply use sctp_bindx to bind to exact local address you need.
You can add or remove local IP's and alter the association. But I'm not sure if you can force a specific path to be used as primary. SCTP basically can use any and switch over to the other.

If you think of something like hot/standby setup, you might want to do this on M3UA level which means having two SCTP associations (two ASPs in one AS)



> On 9 Feb 2024, at 11:05, o.evistel@free.fr wrote:
> 
> 
> Dear Marcelo
> 
> I am using linux-sctp as transport for SIGTRAN M3UA on RHEL 8.4 with multihoming (sctp_bindx(), sctp_connectx() API functions).
> I would like to know, after association setup, if it is possible to instruct SCTP to use a specific local address from the list of bound addresses to reach the peer.
> 
> Regards
> Omar AIT AMRANE
>

Re: Linux SCTP multihoming question

[o.evistel]

Andreas

Do you where I can find the description these ioctl calls.

Regards
Omar
=====================================================================

----- Mail original -----
De: "Andreas Fink" <afink@list.fink.org>
À: "OMAR AIT AMRANE" <o.evistel@free.fr>
Envoyé: Vendredi 9 Février 2024 12:09:48
Objet: Re: Linux SCTP multihoming question

I don't think the SCTP protocol allows this to choose. As it's a hot/hot setup. Multiple path can be used at the same time. But there might be some ioctl calls to force it.


> On 9 Feb 2024, at 12:01, o.evistel@free.fr wrote:
> 
> Hi Andreas,
> 
> My explanation was not clear enough.
> 
> I would like to know if it is possible to force sctp to use alernate address instead of primary address while both of them are available.
> 
> Regards
> Omar 
> =========================================================================================
> 
> ----- Mail original -----
> De: "Andreas Fink" <afink@list.fink.org>
> À: "OMAR AIT AMRANE" <o.evistel@free.fr>
> Cc: "Marcelo Ricardo Leitner" <marcelo.leitner@gmail.com>, "linux-sctp" <linux-sctp@vger.kernel.org>
> Envoyé: Vendredi 9 Février 2024 11:10:22
> Objet: Re: Linux SCTP multihoming question
> 
> yes this is possible. My eSTP does this exactly this way. for M3UA and M2PA You simply use sctp_bindx to bind to exact local address you need.
> You can add or remove local IP's and alter the association. But I'm not sure if you can force a specific path to be used as primary. SCTP basically can use any and switch over to the other.
> 
> If you think of something like hot/standby setup, you might want to do this on M3UA level which means having two SCTP associations (two ASPs in one AS)
> 
> 
> 
>> On 9 Feb 2024, at 11:05, o.evistel@free.fr wrote:
>> 
>> 
>> Dear Marcelo
>> 
>> I am using linux-sctp as transport for SIGTRAN M3UA on RHEL 8.4 with multihoming (sctp_bindx(), sctp_connectx() API functions).
>> I would like to know, after association setup, if it is possible to instruct SCTP to use a specific local address from the list of bound addresses to reach the peer.
>> 
>> Regards
>> Omar AIT AMRANE
>> 
>

RE: Linux SCTP multihoming question

[David Laight]

From: o.evistel@free.fr
> Sent: 09 February 2024 10:06
> 
> I am using linux-sctp as transport for SIGTRAN M3UA on RHEL 8.4 with
> multihoming (sctp_bindx(), sctp_connectx() API functions).
> I would like to know, after association setup, if it is possible to
> instruct SCTP to use a specific local address from the list of bound
> addresses to reach the peer.

Unlikely in the extreme.

If there are 'n' bound local addresses and 'm' remote addresses
(IIRC from the INIT_ACK - but they come from the far end) then
Linux only verifies a route to each local address and picks an
appropriate local address for each one.
So it only sends heartbeats to 'm' addresses, not on 'n * m'
address pairs.

So if anything of this nature did exist it would limit the
remote addresses used, not the local ones.

I think you'll also find that specifying multiple address with
sctp_connectx() only affects where the initial INIT is sent to
(I suspect the addresses get tried in turn).
Once the connection is established the addresses received from
the remote system are almost certainly used instead.

We don't use connectx(), just try a list of addresses in sequence.
This can then contain the addresses of alternate systems as well
and alternate addresses for a single system.

If does have to be said that M3UA works a lot better over TCP!
As well as being a far lighter protocol it is much easier to
get multiple M3UA messages into a single ethernet frame with TCP.

If you've got 10 messages to send, you can use a single send()
to send them (etc). A single recv() can read lots of pending
message and you can cheaply debatch them in userspace.

This all starts mattering when you are trying to send 10000
SMS every second (we don't know what they are sending...)

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: Linux SCTP multihoming question

[Philipp Stanner]

Am Samstag, dem 10.02.2024 um 17:03 +0000 schrieb David Laight:
> From: o.evistel@free.fr
> > Sent: 09 February 2024 10:06
> > 
> > I am using linux-sctp as transport for SIGTRAN M3UA on RHEL 8.4
> > with
> > multihoming (sctp_bindx(), sctp_connectx() API functions).
> > I would like to know, after association setup, if it is possible to
> > instruct SCTP to use a specific local address from the list of
> > bound
> > addresses to reach the peer.
> 
> Unlikely in the extreme.
> 
> If there are 'n' bound local addresses and 'm' remote addresses
> (IIRC from the INIT_ACK - but they come from the far end) then
> Linux only verifies a route to each local address and picks an
> appropriate local address for each one.
> So it only sends heartbeats to 'm' addresses, not on 'n * m'
> address pairs.
> 
> So if anything of this nature did exist it would limit the
> remote addresses used, not the local ones.

I've been told once that using the socket option SO_BINDTODEVICE should
serve the trick, i.e., it should provoke Linux into choosing the picked
device's IP addr as the source IP addr.

I've never tested / verified that, though.
I'm also not sure if it would have other negative consequences, such as
limiting the reachability through non-bound devices.


P.


> 
> I think you'll also find that specifying multiple address with
> sctp_connectx() only affects where the initial INIT is sent to
> (I suspect the addresses get tried in turn).
> Once the connection is established the addresses received from
> the remote system are almost certainly used instead.
> 
> We don't use connectx(), just try a list of addresses in sequence.
> This can then contain the addresses of alternate systems as well
> and alternate addresses for a single system.
> 
> If does have to be said that M3UA works a lot better over TCP!
> As well as being a far lighter protocol it is much easier to
> get multiple M3UA messages into a single ethernet frame with TCP.
> 
> If you've got 10 messages to send, you can use a single send()
> to send them (etc). A single recv() can read lots of pending
> message and you can cheaply debatch them in userspace.
> 
> This all starts mattering when you are trying to send 10000
> SMS every second (we don't know what they are sending...)
> 
> 	David
> 
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes,
> MK1 1PT, UK
> Registration No: 1397386 (Wales)

RE: Linux SCTP multihoming question

[David Laight]

From: Philipp Stanner
> Sent: 10 February 2024 17:09
> 
> Am Samstag, dem 10.02.2024 um 17:03 +0000 schrieb David Laight:
> > From: o.evistel@free.fr
> > > Sent: 09 February 2024 10:06
> > >
> > > I am using linux-sctp as transport for SIGTRAN M3UA on RHEL 8.4
> > > with
> > > multihoming (sctp_bindx(), sctp_connectx() API functions).
> > > I would like to know, after association setup, if it is possible to
> > > instruct SCTP to use a specific local address from the list of
> > > bound
> > > addresses to reach the peer.
> >
> > Unlikely in the extreme.
> >
> > If there are 'n' bound local addresses and 'm' remote addresses
> > (IIRC from the INIT_ACK - but they come from the far end) then
> > Linux only verifies a route to each local address and picks an
> > appropriate local address for each one.
> > So it only sends heartbeats to 'm' addresses, not on 'n * m'
> > address pairs.
> >
> > So if anything of this nature did exist it would limit the
> > remote addresses used, not the local ones.
> 
> I've been told once that using the socket option SO_BINDTODEVICE should
> serve the trick, i.e., it should provoke Linux into choosing the picked
> device's IP addr as the source IP addr.
> 
> I've never tested / verified that, though.
> I'm also not sure if it would have other negative consequences, such as
> limiting the reachability through non-bound devices.

But won't that defeat the entire point of SCTP multihoming?
You need two interfaces in different subnets that use entirely
separate IP networks to connect to the two addresses the remote
system gives you.
(There are really only ever two addresses for each system.)

I don't know what the standards people were smoking, but the
default 'send all my IP addresses to the far end' is so broken.
Requiring the application to know which addresses are 'local'
(typically 192.178.x.x and 10.x.x.x) and pretty much must never
be sent to a remote system (where they can easily mean something
else) is just wrong, it ought to be a property of the protocol stack.

Never mind that the fact that API calls like bindx() were originally
implemented using socket options being embedded in the standard.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: Linux SCTP multihoming question

[Philipp Stanner]

Am Samstag, dem 10.02.2024 um 17:34 +0000 schrieb David Laight:
> From: Philipp Stanner
> > Sent: 10 February 2024 17:09
> > 
> > Am Samstag, dem 10.02.2024 um 17:03 +0000 schrieb David Laight:
> > > From: o.evistel@free.fr
> > > > Sent: 09 February 2024 10:06
> > > > 
> > > > I am using linux-sctp as transport for SIGTRAN M3UA on RHEL 8.4
> > > > with
> > > > multihoming (sctp_bindx(), sctp_connectx() API functions).
> > > > I would like to know, after association setup, if it is
> > > > possible to
> > > > instruct SCTP to use a specific local address from the list of
> > > > bound
> > > > addresses to reach the peer.
> > > 
> > > Unlikely in the extreme.
> > > 
> > > If there are 'n' bound local addresses and 'm' remote addresses
> > > (IIRC from the INIT_ACK - but they come from the far end) then
> > > Linux only verifies a route to each local address and picks an
> > > appropriate local address for each one.
> > > So it only sends heartbeats to 'm' addresses, not on 'n * m'
> > > address pairs.
> > > 
> > > So if anything of this nature did exist it would limit the
> > > remote addresses used, not the local ones.
> > 
> > I've been told once that using the socket option SO_BINDTODEVICE
> > should
> > serve the trick, i.e., it should provoke Linux into choosing the
> > picked
> > device's IP addr as the source IP addr.
> > 
> > I've never tested / verified that, though.
> > I'm also not sure if it would have other negative consequences,
> > such as
> > limiting the reachability through non-bound devices.
> 
> But won't that defeat the entire point of SCTP multihoming?

I guess it defeats the philosophy of the OS with its routing table
being solely responsible for choosing a source address.
I don't see how SCTP would be affected by that, though. SCTP doesn't
demand a certain number of interfaces being available, nor does it make
statements about how the networks behind the interfaces might or might
not be interconnected.

> You need two interfaces in different subnets that use entirely
> separate IP networks to connect to the two addresses the remote
> system gives you.
> (There are really only ever two addresses for each system.)

I assume you're coming from the perspective of a user where SCTP is
utilized with completely redundant and separated networks for
redundancy.
I, however, have only used the protocol in the normal boring Internet –
and there it's definitely possible to reach N endpoints from just 1
outgoing interface. All the endpoints are connected to the Internet,
after all, so the routers will ultimately direct your packages to the
target addr.

> 
> I don't know what the standards people were smoking, but the
> default 'send all my IP addresses to the far end' is so broken.

How else could you implement it?
Only alternative I can think of would be to have some sort of multi-
homing DNS that provides you with several addresses.

P.

> Requiring the application to know which addresses are 'local'
> (typically 192.178.x.x and 10.x.x.x) and pretty much must never
> be sent to a remote system (where they can easily mean something
> else) is just wrong, it ought to be a property of the protocol stack.
> 
> Never mind that the fact that API calls like bindx() were originally
> implemented using socket options being embedded in the standard.
> 
> 	David
> 
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes,
> MK1 1PT, UK
> Registration No: 1397386 (Wales)

RE: Linux SCTP multihoming question

[David Laight]

From: Philipp Stanner
> Sent: 10 February 2024 17:49
...
> > But won't that defeat the entire point of SCTP multihoming?
> 
> I guess it defeats the philosophy of the OS with its routing table
> being solely responsible for choosing a source address.
> I don't see how SCTP would be affected by that, though. SCTP doesn't
> demand a certain number of interfaces being available, nor does it make
> statements about how the networks behind the interfaces might or might
> not be interconnected.

You need to look at why it exists, and what it was trying to replicate.
SCTP is all about carrying telephony signalling over the IP network.
So it is trying (and failing for all sorts of reasons) to give
the same sort of error detection and redundancy as SS7 MTP2 and MTP3.

The normal level of redundancy is to have two separate MTP2 links
(a linkset) to each of two different remote MTP3 systems.
If you are doing it right each MTP2 link is on a separate physical cable.
The SCTP connection is trying to replicate the linkset to a remote
system - so you need two local interfaces connected to two remote ones.
You can have more, but it is basically pointless.

> > You need two interfaces in different subnets that use entirely
> > separate IP networks to connect to the two addresses the remote
> > system gives you.
> > (There are really only ever two addresses for each system.)
> 
> I assume you're coming from the perspective of a user where SCTP is
> utilized with completely redundant and separated networks for
> redundancy.

That is what it is designed for.

> I, however, have only used the protocol in the normal boring Internet –
> and there it's definitely possible to reach N endpoints from just 1
> outgoing interface. All the endpoints are connected to the Internet,
> after all, so the routers will ultimately direct your packages to the
> target addr.

I hope you aren't running any of the SIGTRAM protocols...

> > I don't know what the standards people were smoking, but the
> > default 'send all my IP addresses to the far end' is so broken.
> 
> How else could you implement it?
> Only alternative I can think of would be to have some sort of multi-
> homing DNS that provides you with several addresses.

Given the security in all the SIGTRAN protocols you always used
fixed IP addresses and (in reality) better be using VPN tunnels
if you go anywhere near the public IP network.

You need some property that can be assigned to the local IP / interface
to indicate which ones can be grouped together.
This is already done to exclude localhost.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: Linux SCTP multihoming question

[Philipp Stanner]

Am Samstag, dem 10.02.2024 um 18:02 +0000 schrieb David Laight:
> From: Philipp Stanner
> > Sent: 10 February 2024 17:49
> ...
> > > But won't that defeat the entire point of SCTP multihoming?
> > 
> > I guess it defeats the philosophy of the OS with its routing table
> > being solely responsible for choosing a source address.
> > I don't see how SCTP would be affected by that, though. SCTP
> > doesn't
> > demand a certain number of interfaces being available, nor does it
> > make
> > statements about how the networks behind the interfaces might or
> > might
> > not be interconnected.
> 
> You need to look at why it exists, and what it was trying to
> replicate.
> SCTP is all about carrying telephony signalling over the IP network.
> So it is trying (and failing for all sorts of reasons) to give
> the same sort of error detection and redundancy as SS7 MTP2 and MTP3.
> 
> The normal level of redundancy is to have two separate MTP2 links
> (a linkset) to each of two different remote MTP3 systems.
> If you are doing it right each MTP2 link is on a separate physical
> cable.
> The SCTP connection is trying to replicate the linkset to a remote
> system - so you need two local interfaces connected to two remote
> ones.
> You can have more, but it is basically pointless.
> 
> > > You need two interfaces in different subnets that use entirely
> > > separate IP networks to connect to the two addresses the remote
> > > system gives you.
> > > (There are really only ever two addresses for each system.)
> > 
> > I assume you're coming from the perspective of a user where SCTP is
> > utilized with completely redundant and separated networks for
> > redundancy.
> 
> That is what it is designed for.

All that may be true regarding the original _motivation_ behind the
protocol, but the relevant documents, the RFCs, make no such
statements.
And rightfully so, because a transport protocol should never be tied to
a specific narrow usecase, although people seem to have a hard time
learning that lesson (looking at you, QUIC).
You can even use SCTP as a mere TCP replacement, plainly writing to it
with write(). It's completely up to you. Just as the number of
interfaces you use is.

The protocol even specifies the upper layer protocol number field,
specifically designed to multiplex all sorts of payload protocols (SSH,
HTTP...) over it.


P.

> 
> > I, however, have only used the protocol in the normal boring
> > Internet –
> > and there it's definitely possible to reach N endpoints from just 1
> > outgoing interface. All the endpoints are connected to the
> > Internet,
> > after all, so the routers will ultimately direct your packages to
> > the
> > target addr.
> 
> I hope you aren't running any of the SIGTRAM protocols...
> 
> > > I don't know what the standards people were smoking, but the
> > > default 'send all my IP addresses to the far end' is so broken.
> > 
> > How else could you implement it?
> > Only alternative I can think of would be to have some sort of
> > multi-
> > homing DNS that provides you with several addresses.
> 
> Given the security in all the SIGTRAN protocols you always used
> fixed IP addresses and (in reality) better be using VPN tunnels
> if you go anywhere near the public IP network.
> 
> You need some property that can be assigned to the local IP /
> interface
> to indicate which ones can be grouped together.
> This is already done to exclude localhost.
> 
> 	David
> 
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes,
> MK1 1PT, UK
> Registration No: 1397386 (Wales)