RE: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

RE: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Steve Hill]

Vlad Yasevich wrote on 05 February 2007 16:39:

> Once you start simulating the network failure, how long do you wait?
> 
> If you have not changed rto_max and path_max_retrans, you can end up
> waiting quite a while for the full path switchover.  This will also
> severely slow down your retransmissions...

I'm waiting several minutes, and I'm using setsockopt() to set the
timeouts fairly low:
	srto_initial = 1000ms
	srto_min = 200ms
	srto_max = 1400ms
	spp_pathmaxrxt = 2
	spp_hbinterval = 1000ms

The network's round trip time (indicated by ping) is on the order of
<0.25ms.

 - Steve Hill
   Software Engineer
   Dialogic
   Fordingbridge, Hampshire, UK
   +44-1425-651392
   steve.hill@dialogic.com

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

Hi Steve

Steve Hill wrote:
> Vlad Yasevich wrote on 05 February 2007 16:39:
> 
>> Once you start simulating the network failure, how long do you wait?
>>
>> If you have not changed rto_max and path_max_retrans, you can end up
>> waiting quite a while for the full path switchover.  This will also
>> severely slow down your retransmissions...
> 
> I'm waiting several minutes, and I'm using setsockopt() to set the
> timeouts fairly low:
> 	srto_initial = 1000ms
> 	srto_min = 200ms
> 	srto_max = 1400ms
> 	spp_pathmaxrxt = 2
> 	spp_hbinterval = 1000ms

Ok, thanks for the info.  Two more questions:

  1. What did you set the sinfo_timetolive to?
  2. What specific netfilter rule to do use to simulate network outage?
     I was using '-t filter -A INPUT -i eth0 -p sctp -j DROP'

Just trying to get more info to simulate this.  My prior attempts 
recovered quickly with my patch.

Thanks
-vlad

> 
> The network's round trip time (indicated by ping) is on the order of
> <0.25ms.
> 
>  - Steve Hill
>    Software Engineer
>    Dialogic
>    Fordingbridge, Hampshire, UK
>    +44-1425-651392
>    steve.hill@dialogic.com
> 

RE: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Steve Hill]

Vlad Yasevich wrote on 05 February 2007 20:35:

> would you mind terribly, changing the -d "$net" to the
> -i "$net", and run the script with the interface name instead?

I seem to get the same failure when dropping traffic based on interface
as I do when dropping based on address.

> When I block at the ip address, I see the path failover
> in an odd state.  It looks like it happened, but the flow is
> not resumed.  Receive still doesn't get traffic. I think I might

This sounds like it might be the same problem I'm seeing.

My sender is running the 2.6.16.1 kernel with your patch applied, the
receiver is running Fedora Core 6's 2.6.18-1.2798.fc6 kernel.  The
iptables rules are being set on the receiver (so there should be no odd
interactions between the sender's SCTP stack and iptables - as far as
the sender knows the packets have been transmitted and lost in transit).

Thanks.

 - Steve Hill
   Software Engineer
   Dialogic
   Fordingbridge, Hampshire, UK
   +44-1425-651392
   steve.hill@dialogic.com

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

Steve Hill wrote:
> Vlad Yasevich wrote on 05 February 2007 20:35:
> 
>> would you mind terribly, changing the -d "$net" to the
>> -i "$net", and run the script with the interface name instead?
> 
> I seem to get the same failure when dropping traffic based on interface
> as I do when dropping based on address.

Hmm... can you try with a more recent sender please.  Running 2.6.19 or
2.6.20 with my patch, I don't see this problem when a single interface fails.

I see a full path failover withing the 5 second timeout of the table rule.
Once failover happens, the traffic is using the second interface.

I haven't tried forcing the failover back to the first one, but I can
try flip-flopping them and see what happens.

> 
>> When I block at the ip address, I see the path failover
>> in an odd state.  It looks like it happened, but the flow is
>> not resumed.  Receive still doesn't get traffic. I think I might
> 
> This sounds like it might be the same problem I'm seeing.
> 
> My sender is running the 2.6.16.1 kernel with your patch applied, the
> receiver is running Fedora Core 6's 2.6.18-1.2798.fc6 kernel.  The
> iptables rules are being set on the receiver (so there should be no odd
> interactions between the sender's SCTP stack and iptables - as far as
> the sender knows the packets have been transmitted and lost in transit).

Yes, that's what I am doing as well.  I'll see if I can run a more recent
receiver.

Thanks
-vlad

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

[-- Attachment #1: Type: text/plain, Size: 853 bytes --]

Hi Steve

I think I've tracked this down.  Can you apply the attached patch on top
of the one I posted before and re-run your test.

With both patches, I was able flip-flop the downed interface multiple times
and in all cases path failover completed and data flow resumed.

Here is the modified script I was running:

#!/bin/sh

net1="$1"
net2="$2"

flush() {
    iptables -F
    echo "Flush"
    exit
}

trap flush EXIT

while true; do
    # clear table
    iptables -F
    echo "flushed"
    sleep 5

    # block net1
    iptables -A INPUT -i "$net1" -p sctp -j DROP
    echo "set net1"
    sleep 5

    # clear table
    iptables -F
    echo "flushed"
    sleep 5

    # block net2
    iptables -A INPUT -i "$net2" -p sctp -j DROP
    echo "set net2"
    sleep 5
done

I was able to run this script for 10 minutes sustaining the message flow.

-vlad

[-- Attachment #2: 0001-SCTP-Strike-the-transport-before-updating-rto.patch --]
[-- Type: text/x-patch, Size: 1322 bytes --]

>From 72d6856f7e45a17e0910e0eacd1a01d44fafd1c0 Mon Sep 17 00:00:00 2001
From: Vlad Yasevich <vladislav.yasevich@hp.com>
Date: Wed, 7 Feb 2007 14:58:25 -0500
Subject: [PATCH] [SCTP] Strike the transport before updating rto

Once we reach a poing where we exceed the max.path.retrans
strike the trasport before updating the rto.  This will force
transport switch at the right time, instead of 1 retransmit too
late.

Signed-off-by: Vlad Yasevich <vladislav.yasevich@hp.com>
---
 net/sctp/sm_statefuns.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/net/sctp/sm_statefuns.c b/net/sctp/sm_statefuns.c
index fbbc9e6..801f9d6 100644
--- a/net/sctp/sm_statefuns.c
+++ b/net/sctp/sm_statefuns.c
@@ -4605,12 +4605,12 @@ sctp_disposition_t sctp_sf_do_6_3_3_rtx(const struct sctp_endpoint *ep,
 	 * sent as soon as cwnd allows (normally when a SACK arrives).
 	 */
 
-	/* NB: Rules E4 and F1 are implicit in R1.  */
-	sctp_add_cmd_sf(commands, SCTP_CMD_RETRAN, SCTP_TRANSPORT(transport));
-
 	/* Do some failure management (Section 8.2). */
 	sctp_add_cmd_sf(commands, SCTP_CMD_STRIKE, SCTP_TRANSPORT(transport));
 
+	/* NB: Rules E4 and F1 are implicit in R1.  */
+	sctp_add_cmd_sf(commands, SCTP_CMD_RETRAN, SCTP_TRANSPORT(transport));
+
 	return SCTP_DISPOSITION_CONSUME;
 }
 
-- 
1.5.0.rc3.g6506

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Steve Hill]

On Wed, 7 Feb 2007, Vlad Yasevich wrote:

> I think I've tracked this down.  Can you apply the attached patch on top
> of the one I posted before and re-run your test.

Using the 2.6.20 kernel on the sending side with both patches applied, the
problem seems to be fixed.

Thanks.

-- 
 - Steve Hill
   Software Engineer
   Dialogic
   Fordingbridge, Hampshire, UK
   +44-1425-651392
   steve.hill@dialogic.com

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

Steve Hill wrote:
> On Wed, 7 Feb 2007, Vlad Yasevich wrote:
> 
>> I think I've tracked this down.  Can you apply the attached patch on top
>> of the one I posted before and re-run your test.
> 
> Using the 2.6.20 kernel on the sending side with both patches applied, the
> problem seems to be fixed.
> 
> Thanks.
> 

Excellent.  I'll push out the new ones to Sridhar today.

-vlad

RE: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Steve Hill]

Vlad Yasevich wrote on 05 February 2007 17:08:

>   1. What did you set the sinfo_timetolive to?

I presume you mean the timetolive parameter of sctp_sendmsg()? - this
was set to 1400ms (as previously mentioned, this was in error but it
does appear to have highlighted a problem with the stack itself).

>   2. What specific netfilter rule to do use to simulate
> network outage?
>      I was using '-t filter -A INPUT -i eth0 -p sctp -j DROP'

iptables -A INPUT -d 192.168.2.0/24 -p sctp -j DROP

> Just trying to get more info to simulate this.  My prior attempts
> recovered quickly with my patch.

I usually (but not always - sometimes it happens on the first attempt)
have to add and remove the iptables rule a few times while running
traffic over the association in order to reproduce the problem.  I'm
running traffic at a rate of around 500 data chunks per second.  Each
data chunk has a 44 octet payload.

The script I'm using to toggle the iptables rule is below:

----------
#!/bin/sh

net="$1"

flush() {
	iptables -F
	echo "Flush"
	exit
}

trap flush EXIT

while true; do
	iptables -A INPUT -d "$net" -p sctp -j DROP
	echo "set"
	sleep 5
	iptables -F
	echo "flushed"
	sleep 5
done 
----------

 - Steve Hill
   Software Engineer
   Dialogic
   Fordingbridge, Hampshire, UK
   +44-1425-651392
   steve.hill@dialogic.com

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

Hi Steve

would you mind terribly, changing the -d "$net" to the 
-i "$net", and run the script with the interface name instead?

The reason is, that I see 2 different behaviors between blocking
by interface and blocking by IP and would like to find out if
you see it too.

When I block at the interface ( -i eth1 in my case), I see
the path failover happen and flow if resumed.

When I block at the ip address, I see the path failover
in an odd state.  It looks like it happened, but the flow is
not resumed.  Receive still doesn't get traffic. I think I might
be running a buggy receiver, but I am not 100% sure.  In my
case, the sender if running 2.6.10-rc7 and receiver is running
Ubuntu 2.6.17-10.

I'll try running against a different receiver as well.

-vlad

Steve Hill wrote:
> Vlad Yasevich wrote on 05 February 2007 17:08:
> 
>>   1. What did you set the sinfo_timetolive to?
> 
> I presume you mean the timetolive parameter of sctp_sendmsg()? - this
> was set to 1400ms (as previously mentioned, this was in error but it
> does appear to have highlighted a problem with the stack itself).
> 
>>   2. What specific netfilter rule to do use to simulate
>> network outage?
>>      I was using '-t filter -A INPUT -i eth0 -p sctp -j DROP'
> 
> iptables -A INPUT -d 192.168.2.0/24 -p sctp -j DROP
> 
>> Just trying to get more info to simulate this.  My prior attempts
>> recovered quickly with my patch.
> 
> I usually (but not always - sometimes it happens on the first attempt)
> have to add and remove the iptables rule a few times while running
> traffic over the association in order to reproduce the problem.  I'm
> running traffic at a rate of around 500 data chunks per second.  Each
> data chunk has a 44 octet payload.
> 
> The script I'm using to toggle the iptables rule is below:
> 
> ----------
> #!/bin/sh
> 
> net="$1"
> 
> flush() {
> 	iptables -F
> 	echo "Flush"
> 	exit
> }
> 
> trap flush EXIT
> 
> while true; do
> 	iptables -A INPUT -d "$net" -p sctp -j DROP
> 	echo "set"
> 	sleep 5
> 	iptables -F
> 	echo "flushed"
> 	sleep 5
> done 
> ----------
> 
>  - Steve Hill
>    Software Engineer
>    Dialogic
>    Fordingbridge, Hampshire, UK
>    +44-1425-651392
>    steve.hill@dialogic.com
> 

RE: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Steve Hill]

Vlad Yasevich wrote on 25 January 2007 16:33:

> Can you try the attached patch and let me know if the problem is
> fixed.  You can try reducing rto_max or path_max_retrans to get the
> failover to happen a little faster. 

Sorry for the delay - I've been on vacation for the past week.

I've tried applying the patch.  However, the failure still seems to
happen in the original test system with a patched kernel.  A look at the
network traffic shows that the receiving side is still returning a
gap-ack, the chunks in the gap are never resent and I don't see a
FORWARD TSN for them.

 - Steve Hill
   Software Engineer
   Dialogic
   Fordingbridge, Hampshire, UK
   +44-1425-651392
   steve.hill@dialogic.com

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

Steve Hill wrote:
> Vlad Yasevich wrote on 25 January 2007 16:33:
> 
>> Can you try the attached patch and let me know if the problem is
>> fixed.  You can try reducing rto_max or path_max_retrans to get the
>> failover to happen a little faster. 
> 
> Sorry for the delay - I've been on vacation for the past week.
> 
> I've tried applying the patch.  However, the failure still seems to
> happen in the original test system with a patched kernel.  A look at the
> network traffic shows that the receiving side is still returning a
> gap-ack, the chunks in the gap are never resent and I don't see a
> FORWARD TSN for them.

Once you start simulating the network failure, how long do you wait?

If you have not changed rto_max and path_max_retrans, you can end up
waiting quite a while for the full path switchover.  This will also
severely slow down your retransmissions...

Try reducing /proc/sys/net/sctp/rto_max to say 6000ms (6 seconds).  It
defaults to (60000 ms or 1 minute).  That reduction means that the
retransmit timeout will max out at 6 seconds.

The problem is that you will not see the FORWARD TSN on the alternate
path until the original path is marked down.  That requires us to
exceed the path_max_retrans counter
(/proc/sys/net/sctp/path_max_retrans).  This means that you will see
the gap grow until a time when the rto exceeds the data expiry time.
At this pint, you will not see any new data sent, since it will expire
before being sent.  So, it will look to you like nothing is happening.

The gap will be closed once the FWD-TSN is transmitted over the
alternate transport.

-vlad

> 
>  - Steve Hill
>    Software Engineer
>    Dialogic
>    Fordingbridge, Hampshire, UK
>    +44-1425-651392
>    steve.hill@dialogic.com
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier.
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Lksctp-developers mailing list
> Lksctp-developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/lksctp-developers
> 

Fw: Intermittent SCTP multihoming breakage

[Andrew Morton]

Begin forwarded message:

Date: Wed, 3 Jan 2007 11:54:26 +0000
From: Steve Hill <steve.hill@dialogic.com>
To: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Subject: Intermittent SCTP multihoming breakage



Apologies if I'm posting to the wrong list - the lksctp lists seem to be a
bit dead these days and a bit of Googling seemed to inidicate that SCTP
developemnt discussions might have moved here.

I'm running under the 2.6.16.1 kernel and have an intermittent problem
with the SCTP stack.  Having reviewed the git logs I can't see any
indication that the problem has been fixed in more recent kernels, but it
is very difficult to test since it is so intermittent.

I am running a multihomed connection between 2 machines, (2 NICs on
each machine, so 2 paths for the connection) and tcpdump shows heartbeat
requests and acks on both paths.  Putting data over the link correctly
sends it over the first path.

If I drop the traffic on one of the NICs then most of the time it
correctly fails over the the second path and I see the data being sent
and acknowledged correctly on the second path.  However, I also
intermittently see two failure conditions:

1. Sometimes, just after failing over to the second path I see an ABORT.
2. More frequently, the association stays up indefinately, with heartbeat
requests and acks on the second path, but no data chunks are sent even
though the transmit queue on the transmitting end appears to be full and
the socket is blocking writes.

I have been adding debugging to the kernel in an attempt to track down the
source of the second failure condition, and I am wondering if anyone else
has seen similar behaviour?

-- 
 - Steve Hill
   Software Engineer
   Dialogic
   Fordingbridge, Hampshire, UK
   +44-1425-651392
   steve.hill@dialogic.com
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: Fw: Intermittent SCTP multihoming breakage

[Sridhar Samudrala]

On Wed, 2007-01-03 at 15:46 -0800, Andrew Morton wrote:
> 
> Begin forwarded message:
> 
> Date: Wed, 3 Jan 2007 11:54:26 +0000
> From: Steve Hill <steve.hill@dialogic.com>
> To: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
> Subject: Intermittent SCTP multihoming breakage
> 
> 
> 
> Apologies if I'm posting to the wrong list - the lksctp lists seem to be a
> bit dead these days and a bit of Googling seemed to inidicate that SCTP
> developemnt discussions might have moved here.

No. lksctp-developers mailing list is still the best place for SCTP related
discussions. You can subscribe and look in the archives at 
  http://lists.sourceforge.net/lists/listinfo/lksctp-developers

> 
> I'm running under the 2.6.16.1 kernel and have an intermittent problem
> with the SCTP stack.  Having reviewed the git logs I can't see any
> indication that the problem has been fixed in more recent kernels, but it
> is very difficult to test since it is so intermittent.

If possible, i would suggest moving to the latest mainline 2.6.19.
But 2.6.16.1 should work OK for simple multihoming cases.

> 
> I am running a multihomed connection between 2 machines, (2 NICs on
> each machine, so 2 paths for the connection) and tcpdump shows heartbeat
> requests and acks on both paths.  Putting data over the link correctly
> sends it over the first path.
How are the 2 machines connected? Are they connected directly or
via a router?

Do you see both the addresses when you do cat /proc/net/sctp/assocs 
after the association is established on both the peers?

> 
> If I drop the traffic on one of the NICs then most of the time it
> correctly fails over the the second path and I see the data being sent
> and acknowledged correctly on the second path.  However, I also
> intermittently see two failure conditions:

How are you dropping traffic? You could try simulating failover by
bringing down the interface or physically removing the link.

> 
> 1. Sometimes, just after failing over to the second path I see an ABORT.
This seems to indicate that somehow the app has terminated.

> 2. More frequently, the association stays up indefinately, with heartbeat
> requests and acks on the second path, but no data chunks are sent even
> though the transmit queue on the transmitting end appears to be full and
> the socket is blocking writes.
This is strange. Can you collect tcpdump traces on sender and receiver when 
this happens?

Thanks
Sridhar

> 
> I have been adding debugging to the kernel in an attempt to track down the
> source of the second failure condition, and I am wondering if anyone else
> has seen similar behaviour?
> 
> --
>  - Steve Hill
>    Software Engineer
>    Dialogic
>    Fordingbridge, Hampshire, UK
>    +44-1425-651392
>    steve.hill@dialogic.com
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Fw: Intermittent SCTP multihoming breakage

[Steve Hill]

On Wed, 3 Jan 2007, Sridhar Samudrala wrote:

Sorry for the delay in replying.

> No. lksctp-developers mailing list is still the best place for SCTP related
> discussions. You can subscribe and look in the archives at
>   http://lists.sourceforge.net/lists/listinfo/lksctp-developers

Hmm, I had a look there and it seemed reasonably inactive and overrun by
spam.. (And I've been unable to subscribe).

> How are the 2 machines connected? Are they connected directly or
> via a router?

They are currently connected together directly through crossover cables.

> Do you see both the addresses when you do cat /proc/net/sctp/assocs
> after the association is established on both the peers?

Yes, the contents of /proc/net/sctp/assocs looks correct.

> How are you dropping traffic? You could try simulating failover by
> bringing down the interface or physically removing the link.

I have been using iptables to drop SCTP packets on both the INPUT and
OUTPUT chains.  However, I get the same results if I just unplug the
network cable (using iptables is easier for my testing since I don't have
to crawl around behind the test systems :)

> > 1. Sometimes, just after failing over to the second path I see an ABORT.
> This seems to indicate that somehow the app has terminated.

The abort _appears_ to be caused by a retransmit timer expiring, causing
the SCTP stack to tear down the association.  However, I haven't done much
investigation of this problem yet - I've been focussing on the second
problem since it seems to happen more frequently.

> > 2. More frequently, the association stays up indefinately, with heartbeat
> > requests and acks on the second path, but no data chunks are sent even
> > though the transmit queue on the transmitting end appears to be full and
> > the socket is blocking writes.
> This is strange. Can you collect tcpdump traces on sender and receiver when
> this happens?

I've taken dumps of the data on the wire for both paths:
  http://www.nexusuk.org/~steve/sctp/path1.pcap
  http://www.nexusuk.org/~steve/sctp/path2.pcap

I can't see anything odd in the network traffic - it just stops as if it
has no more data to send.  However, the socket appears to still be
blocking so the application cannot give it any new data.

This seems to be a problem with the abandonment functionality:
1. Transmit chunk 1.  The transmitted list now contains chunk 1.
2. Chunk 1 and it's retransmissions get lost on the network.
3. Abandon chunk 1.  The transmitted list is now empty.
4. Transmit chunk 2.  the transmitted list now contains chunk 2
5. Receive a gap-ack for chunk 2, indicating that chunk 1 is missing.
At this point, the T3 timer is disabled at the bottom of
sctp_check_transmitted() since all the chunks in the transmitted queue are
gap-acked.  The whole connection now stalls, waiting for the SACK for
chunk 1 that will never arrive.

It should be noted that this is not unordered data and I'm not clear on
how abandoned chunks are supposed to be handled - I hadn't intentionally
enabled the abandonment functionality, the timetolive was set on the
transmitted chunks by accident.

-- 
 - Steve Hill
   Software Engineer
   Dialogic
   Fordingbridge, Hampshire, UK
   +44-1425-651392
   steve.hill@dialogic.com

Re: Fw: Intermittent SCTP multihoming breakage

[Sridhar Samudrala]

On Wed, 2007-01-10 at 11:55 +0000, Steve Hill wrote:
> On Wed, 3 Jan 2007, Sridhar Samudrala wrote:
> 
> Sorry for the delay in replying.
> 
> > No. lksctp-developers mailing list is still the best place for SCTP related
> > discussions. You can subscribe and look in the archives at
> >   http://lists.sourceforge.net/lists/listinfo/lksctp-developers
> 
> Hmm, I had a look there and it seemed reasonably inactive and overrun by
> spam.. (And I've been unable to subscribe).

It is not very active all the time, but you can see bursts of activity. 
All the SCTP patches and issues get discussed there before being 
submitted to netdev.
I didn't realize that so much spam is getting on this list. I don't see
any of it as my mailer's spam filters seem to be pretty effective. I will
see if there is a way to set up spam filters on sourceforge lists.

I haven't heard of any issues with subscription. Could you try again and
let me know offline if you still have issues.

> 
> > How are the 2 machines connected? Are they connected directly or
> > via a router?
> 
> They are currently connected together directly through crossover cables.
> 
> > Do you see both the addresses when you do cat /proc/net/sctp/assocs
> > after the association is established on both the peers?
> 
> Yes, the contents of /proc/net/sctp/assocs looks correct.
> 
> > How are you dropping traffic? You could try simulating failover by
> > bringing down the interface or physically removing the link.
> 
> I have been using iptables to drop SCTP packets on both the INPUT and
> OUTPUT chains.  However, I get the same results if I just unplug the
> network cable (using iptables is easier for my testing since I don't have
> to crawl around behind the test systems :)
> 
> > > 1. Sometimes, just after failing over to the second path I see an ABORT.
> > This seems to indicate that somehow the app has terminated.
> 
> The abort _appears_ to be caused by a retransmit timer expiring, causing
> the SCTP stack to tear down the association.  However, I haven't done much
> investigation of this problem yet - I've been focussing on the second
> problem since it seems to happen more frequently.
> 
> > > 2. More frequently, the association stays up indefinately, with heartbeat
> > > requests and acks on the second path, but no data chunks are sent even
> > > though the transmit queue on the transmitting end appears to be full and
> > > the socket is blocking writes.
> > This is strange. Can you collect tcpdump traces on sender and receiver when
> > this happens?
> 
> I've taken dumps of the data on the wire for both paths:
>   http://www.nexusuk.org/~steve/sctp/path1.pcap
>   http://www.nexusuk.org/~steve/sctp/path2.pcap
> 
> I can't see anything odd in the network traffic - it just stops as if it
> has no more data to send.  However, the socket appears to still be
> blocking so the application cannot give it any new data.
> 
> This seems to be a problem with the abandonment functionality:
> 1. Transmit chunk 1.  The transmitted list now contains chunk 1.
> 2. Chunk 1 and it's retransmissions get lost on the network.
> 3. Abandon chunk 1.  The transmitted list is now empty.
> 4. Transmit chunk 2.  the transmitted list now contains chunk 2
> 5. Receive a gap-ack for chunk 2, indicating that chunk 1 is missing.
> At this point, the T3 timer is disabled at the bottom of
> sctp_check_transmitted() since all the chunks in the transmitted queue are
> gap-acked.  The whole connection now stalls, waiting for the SACK for
> chunk 1 that will never arrive.
> 
> It should be noted that this is not unordered data and I'm not clear on
> how abandoned chunks are supposed to be handled - I hadn't intentionally
> enabled the abandonment functionality, the timetolive was set on the
> transmitted chunks by accident.

So looks like there may be an issue with PR-SCTP(partial reliability)
support and packet loss. I will take a look into this.

Do you still see this problem even if you don't set timetolive?
If you don't need chunks to be expired, you can disable pr-sctp using 
echo 0 > /proc/sys/net/sctp/prsctp-enable

Thanks
Sridhar

Re: Fw: Intermittent SCTP multihoming breakage

[Steve Hill]

On Wed, 10 Jan 2007, Sridhar Samudrala wrote:

> So looks like there may be an issue with PR-SCTP(partial reliability)
> support and packet loss. I will take a look into this.
>
> Do you still see this problem even if you don't set timetolive?

No, the problem seems to go away if the timetolive is set to 0, so this is
what I have now done since I had not intended to set the timetolive in the
first place (but I thought it was still worth posting details of the
problem since it does appear to be a bug).

-- 
 - Steve Hill
   Software Engineer
   Dialogic
   Fordingbridge, Hampshire, UK
   +44-1425-651392
   steve.hill@dialogic.com

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

[-- Attachment #1: Type: text/plain, Size: 1190 bytes --]

Hi Steve

Steve Hill wrote:
> On Wed, 10 Jan 2007, Sridhar Samudrala wrote:
> 
>> So looks like there may be an issue with PR-SCTP(partial reliability)
>> support and packet loss. I will take a look into this.
>>
>> Do you still see this problem even if you don't set timetolive?
> 
> No, the problem seems to go away if the timetolive is set to 0, so this is
> what I have now done since I had not intended to set the timetolive in the
> first place (but I thought it was still worth posting details of the
> problem since it does appear to be a bug).
> 

I think I found this bug.  It was rather interesting to figure out.  The problem
appears to be that data messages time-out within the rto.  As a result, they
move the abandoned list and are never retransmitted.  This clears the retransmit
list and the retransmit timer, however the data is still charged as in-flight against
the association.  This in turn causes new data not to be send, since we are 'supposedly'
utilizing our congestion window.

Can you try the attached patch and let me know if the problem is fixed.  You can 
try reducing rto_max or path_max_retrans to get the failover to happen a little faster.

Regards
-vlad

[-- Attachment #2: 0001-SCTP-Fix-connection-hang-slowdown-with-PR-SCTP.txt --]
[-- Type: text/plain, Size: 2804 bytes --]

[SCTP]: Fix connection hang with PR-SCTP

The problem that this patch corrects happens when all
of the following conditions are satisfisfied:
    1.  PR-SCTP is used and the timeout on the chunks is
        set below RTO.Max.
    2.  One of the paths on a multihomed associations
        is brought down.

In this scenario, data will expire within the rto of the
initial transmission and will never be retransmitted.  However
this data still fills the send buffer and is counted against
the association as outstanding data.  This causes any new
data to not be sent and retransmission to not happen.

The fix is to discount the abandoned data from the outstanding
count and peers rwnd estimation.  This allows new data to
be sent and a retransmission timer restarted.  Even though
this new data will most like expire withing the rto, the
timer still counts as a strike agains the transport and forces
the FORWARD-TSN chunk to be retransmitted as well.

Signed-off-by: Vlad Yasevich <vladislav.yasevich@hp.com>
---
 net/sctp/outqueue.c |   27 ++++++++++++++++++++++-----
 1 files changed, 22 insertions(+), 5 deletions(-)

diff --git a/net/sctp/outqueue.c b/net/sctp/outqueue.c
index fba567a..54d1b7f 100644
--- a/net/sctp/outqueue.c
+++ b/net/sctp/outqueue.c
@@ -396,6 +396,19 @@ void sctp_retransmit_mark(struct sctp_outq *q,
 		if (sctp_chunk_abandoned(chunk)) {
 			list_del_init(lchunk);
 			sctp_insert_list(&q->abandoned, lchunk);
+
+			/* If this chunk has not been previousely acked,
+			 * stop considering it 'outstanding'.  Our peer
+			 * will most likely never see it since it will
+			 * not be retransmitted
+			 */
+			if (!chunk->tsn_gap_acked) {
+				chunk->transport->flight_size -=
+						sctp_data_size(chunk);
+				q->outstanding_bytes -= sctp_data_size(chunk);
+				q->asoc->peer.rwnd += (sctp_data_size(chunk) +
+							sizeof(struct sk_buff));
+			}
 			continue;
 		}
 
@@ -1244,6 +1257,15 @@ static void sctp_check_transmitted(struct sctp_outq *q,
 		if (sctp_chunk_abandoned(tchunk)) {
 			/* Move the chunk to abandoned list. */
 			sctp_insert_list(&q->abandoned, lchunk);
+
+			/* If this chunk has not been acked, stop
+			 * considering it as 'outstanding'.
+			 */
+			if (!tchunk->tsn_gap_acked) {
+				tchunk->transport->flight_size -=
+						sctp_data_size(tchunk);
+				q->outstanding_bytes -= sctp_data_size(tchunk);
+			}
 			continue;
 		}
 
@@ -1695,11 +1717,6 @@ static void sctp_generate_fwdtsn(struct sctp_outq *q, __u32 ctsn)
 		 */ 
 		if (TSN_lte(tsn, ctsn)) {
 			list_del_init(lchunk);
-			if (!chunk->tsn_gap_acked) {
-				chunk->transport->flight_size -=
-					sctp_data_size(chunk);
-				q->outstanding_bytes -= sctp_data_size(chunk);
-			}
 			sctp_chunk_free(chunk);
 		} else {
 			if (TSN_lte(tsn, asoc->adv_peer_ack_point+1)) {
-- 
1.4.4.2.g8336

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

BTW, if anyone needs a reproducer, I can provide one.

-vlad