[PATCH 1/4] sctp: fix association hangs due to off-by-one errors in sctp_tsnmap_grow()

[PATCH 1/4] sctp: fix association hangs due to off-by-one errors in sctp_tsnmap_grow()

[Roberts, Lee A.]

From: Lee A. Roberts <lee.roberts@hp.com>

Resolve SCTP association hangs observed during SCTP stress
testing.  Observable symptoms include communications hangs
with data being held in the association lobby (ordering)
queue.  Close examination of reassembly/ordering queues shows
duplicated packets.

In sctp_tsnmap_grow(), correct off-by-one errors when copying
and resizing the tsnmap.  If max_tsn_seen is in the LSB of the
word, this bit can be lost, causing the corresponding packet
to be transmitted again and to be entered as a duplicate into
the SCTP reassembly/ordering queues.

Patch applies to linux-3.8 kernel.

Signed-off-by: Lee A. Roberts <lee.roberts@hp.com>
---
 net/sctp/tsnmap.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff -uprN -X linux-3.8-vanilla/Documentation/dontdiff linux-3.8-vanilla/net/sctp/tsnmap.c linux-3.8-SCTP+1/net/sctp/tsnmap.c
--- linux-3.8-vanilla/net/sctp/tsnmap.c	2013-02-18 16:58:34.000000000 -0700
+++ linux-3.8-SCTP+1/net/sctp/tsnmap.c	2013-02-20 08:01:02.555223259 -0700
@@ -369,14 +369,15 @@ static int sctp_tsnmap_grow(struct sctp_
 	if (gap >= SCTP_TSN_MAP_SIZE)
 		return 0;
 
-	inc = ALIGN((gap - map->len),BITS_PER_LONG) + SCTP_TSN_MAP_INCREMENT;
+	inc = ALIGN((gap - map->len + 1), BITS_PER_LONG)
+		+ SCTP_TSN_MAP_INCREMENT;
 	len = min_t(u16, map->len + inc, SCTP_TSN_MAP_SIZE);
 
 	new = kzalloc(len>>3, GFP_ATOMIC);
 	if (!new)
 		return 0;
 
-	bitmap_copy(new, map->tsn_map, map->max_tsn_seen - map->base_tsn);
+	bitmap_copy(new, map->tsn_map, map->max_tsn_seen - map->base_tsn + 1);
 	kfree(map->tsn_map);
 	map->tsn_map = new;
 	map->len = len;

Re: [PATCH 1/4] sctp: fix association hangs due to off-by-one errors in sctp_tsnmap_grow()

[Vlad Yasevich]

On 02/21/2013 11:44 AM, Roberts, Lee A. wrote:
> From: Lee A. Roberts <lee.roberts@hp.com>
>
> Resolve SCTP association hangs observed during SCTP stress
> testing.  Observable symptoms include communications hangs
> with data being held in the association lobby (ordering)
> queue.  Close examination of reassembly/ordering queues shows
> duplicated packets.
>
> In sctp_tsnmap_grow(), correct off-by-one errors when copying
> and resizing the tsnmap.  If max_tsn_seen is in the LSB of the
> word, this bit can be lost, causing the corresponding packet
> to be transmitted again and to be entered as a duplicate into
> the SCTP reassembly/ordering queues.
>
> Patch applies to linux-3.8 kernel.
>
> Signed-off-by: Lee A. Roberts <lee.roberts@hp.com>
> ---
>   net/sctp/tsnmap.c |    5 +++--
>   1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff -uprN -X linux-3.8-vanilla/Documentation/dontdiff linux-3.8-vanilla/net/sctp/tsnmap.c linux-3.8-SCTP+1/net/sctp/tsnmap.c
> --- linux-3.8-vanilla/net/sctp/tsnmap.c	2013-02-18 16:58:34.000000000 -0700
> +++ linux-3.8-SCTP+1/net/sctp/tsnmap.c	2013-02-20 08:01:02.555223259 -0700
> @@ -369,14 +369,15 @@ static int sctp_tsnmap_grow(struct sctp_
>   	if (gap >= SCTP_TSN_MAP_SIZE)

No that I think about this a bit more, this should be gap + 1.  If you 
do that, you might as well call sctp_tsnmap_grow() with gap+1 as 
argument and then can just use the 'gap' everywhere inside.

>   		return 0;
>
> -	inc = ALIGN((gap - map->len),BITS_PER_LONG) + SCTP_TSN_MAP_INCREMENT;
> +	inc = ALIGN((gap - map->len + 1), BITS_PER_LONG)
> +		+ SCTP_TSN_MAP_INCREMENT;
>   	len = min_t(u16, map->len + inc, SCTP_TSN_MAP_SIZE);
>
>   	new = kzalloc(len>>3, GFP_ATOMIC);
>   	if (!new)
>   		return 0;
>
> -	bitmap_copy(new, map->tsn_map, map->max_tsn_seen - map->base_tsn);
> +	bitmap_copy(new, map->tsn_map, map->max_tsn_seen - map->base_tsn + 1);

Can simplify that this by using map->cumulative_tsn_ack_point instead of 
base_tsn.

-vlad

>   	kfree(map->tsn_map);
>   	map->tsn_map = new;
>   	map->len = len;
> --
> To unsubscribe from this list: send the line "unsubscribe linux-sctp" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: [PATCH 1/4] sctp: fix association hangs due to off-by-one errors in sctp_tsnmap_grow()

[Roberts, Lee A.]

Vlad,

> -----Original Message-----
> From: Vlad Yasevich [mailto:vyasevich@gmail.com]
> Sent: Thursday, February 21, 2013 10:19 AM
> To: Roberts, Lee A.
> Cc: linux-sctp@vger.kernel.org; netdev@vger.kernel.org; linux-kernel@vger.kernel.org
> Subject: Re: [PATCH 1/4] sctp: fix association hangs due to off-by-one errors in sctp_tsnmap_grow()
> 
> On 02/21/2013 11:44 AM, Roberts, Lee A. wrote:
> > From: Lee A. Roberts <lee.roberts@hp.com>
> >
> > Resolve SCTP association hangs observed during SCTP stress
> > testing.  Observable symptoms include communications hangs
> > with data being held in the association lobby (ordering)
> > queue.  Close examination of reassembly/ordering queues shows
> > duplicated packets.
> >
> > In sctp_tsnmap_grow(), correct off-by-one errors when copying
> > and resizing the tsnmap.  If max_tsn_seen is in the LSB of the
> > word, this bit can be lost, causing the corresponding packet
> > to be transmitted again and to be entered as a duplicate into
> > the SCTP reassembly/ordering queues.
> >
> > Patch applies to linux-3.8 kernel.
> >
> > Signed-off-by: Lee A. Roberts <lee.roberts@hp.com>
> > ---
> >   net/sctp/tsnmap.c |    5 +++--
> >   1 file changed, 3 insertions(+), 2 deletions(-)
> >
> > diff -uprN -X linux-3.8-vanilla/Documentation/dontdiff linux-3.8-vanilla/net/sctp/tsnmap.c linux-
> 3.8-SCTP+1/net/sctp/tsnmap.c
> > --- linux-3.8-vanilla/net/sctp/tsnmap.c	2013-02-18 16:58:34.000000000 -0700
> > +++ linux-3.8-SCTP+1/net/sctp/tsnmap.c	2013-02-20 08:01:02.555223259 -0700
> > @@ -369,14 +369,15 @@ static int sctp_tsnmap_grow(struct sctp_
> >   	if (gap >= SCTP_TSN_MAP_SIZE)
> 
> No that I think about this a bit more, this should be gap + 1.  If you
> do that, you might as well call sctp_tsnmap_grow() with gap+1 as
> argument and then can just use the 'gap' everywhere inside.

I think the calculation of "gap" in sctp_tsnmap_mark() should change:

-	gap = tsn - map->base_tsn;
+	gap = tsn - map->cumulative_tsn_ack_point;

> 
> >   		return 0;
> >
> > -	inc = ALIGN((gap - map->len),BITS_PER_LONG) + SCTP_TSN_MAP_INCREMENT;
> > +	inc = ALIGN((gap - map->len + 1), BITS_PER_LONG)
> > +		+ SCTP_TSN_MAP_INCREMENT;
> >   	len = min_t(u16, map->len + inc, SCTP_TSN_MAP_SIZE);
> >
> >   	new = kzalloc(len>>3, GFP_ATOMIC);
> >   	if (!new)
> >   		return 0;
> >
> > -	bitmap_copy(new, map->tsn_map, map->max_tsn_seen - map->base_tsn);
> > +	bitmap_copy(new, map->tsn_map, map->max_tsn_seen - map->base_tsn + 1);
> 
> Can simplify that this by using map->cumulative_tsn_ack_point instead of
> base_tsn.
> 
> -vlad
> 

I changed the code to use "cumulative_tsn_ack_point" in an updated version
of the patch.

                                         -- Lee

> >   	kfree(map->tsn_map);
> >   	map->tsn_map = new;
> >   	map->len = len;
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-sctp" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >

RE: [PATCH 1/4] sctp: fix association hangs due to off-by-one errors in sctp_tsnmap_grow()

[Roberts, Lee A.]

Vlad,

> -----Original Message-----
> From: Roberts, Lee A.
> Sent: Thursday, February 21, 2013 11:00 AM
> To: 'Vlad Yasevich'
> Cc: linux-sctp@vger.kernel.org; netdev@vger.kernel.org; linux-kernel@vger.kernel.org
> Subject: RE: [PATCH 1/4] sctp: fix association hangs due to off-by-one errors in sctp_tsnmap_grow()
> 
> Vlad,
> 
> > -----Original Message-----
> > From: Vlad Yasevich [mailto:vyasevich@gmail.com]
> > Sent: Thursday, February 21, 2013 10:19 AM
> > To: Roberts, Lee A.
> > Cc: linux-sctp@vger.kernel.org; netdev@vger.kernel.org; linux-kernel@vger.kernel.org
> > Subject: Re: [PATCH 1/4] sctp: fix association hangs due to off-by-one errors in sctp_tsnmap_grow()
> >
> > On 02/21/2013 11:44 AM, Roberts, Lee A. wrote:
> > > From: Lee A. Roberts <lee.roberts@hp.com>
> > >
> > > Resolve SCTP association hangs observed during SCTP stress
> > > testing.  Observable symptoms include communications hangs
> > > with data being held in the association lobby (ordering)
> > > queue.  Close examination of reassembly/ordering queues shows
> > > duplicated packets.
> > >
> > > In sctp_tsnmap_grow(), correct off-by-one errors when copying
> > > and resizing the tsnmap.  If max_tsn_seen is in the LSB of the
> > > word, this bit can be lost, causing the corresponding packet
> > > to be transmitted again and to be entered as a duplicate into
> > > the SCTP reassembly/ordering queues.
> > >
> > > Patch applies to linux-3.8 kernel.
> > >
> > > Signed-off-by: Lee A. Roberts <lee.roberts@hp.com>
> > > ---
> > >   net/sctp/tsnmap.c |    5 +++--
> > >   1 file changed, 3 insertions(+), 2 deletions(-)
> > >
> > > diff -uprN -X linux-3.8-vanilla/Documentation/dontdiff linux-3.8-vanilla/net/sctp/tsnmap.c linux-
> > 3.8-SCTP+1/net/sctp/tsnmap.c
> > > --- linux-3.8-vanilla/net/sctp/tsnmap.c	2013-02-18 16:58:34.000000000 -0700
> > > +++ linux-3.8-SCTP+1/net/sctp/tsnmap.c	2013-02-20 08:01:02.555223259 -0700
> > > @@ -369,14 +369,15 @@ static int sctp_tsnmap_grow(struct sctp_
> > >   	if (gap >= SCTP_TSN_MAP_SIZE)
> >
> > No that I think about this a bit more, this should be gap + 1.  If you
> > do that, you might as well call sctp_tsnmap_grow() with gap+1 as
> > argument and then can just use the 'gap' everywhere inside.
> 
> I think the calculation of "gap" in sctp_tsnmap_mark() should change:
> 
> -	gap = tsn - map->base_tsn;
> +	gap = tsn - map->cumulative_tsn_ack_point;
> 

Oops, this breaks the logic that follows.  Using "gap + 1" in the call should work.
I sent an updated patch (v3).

                                    - Lee

> >
> > >   		return 0;
> > >
> > > -	inc = ALIGN((gap - map->len),BITS_PER_LONG) + SCTP_TSN_MAP_INCREMENT;
> > > +	inc = ALIGN((gap - map->len + 1), BITS_PER_LONG)
> > > +		+ SCTP_TSN_MAP_INCREMENT;
> > >   	len = min_t(u16, map->len + inc, SCTP_TSN_MAP_SIZE);
> > >
> > >   	new = kzalloc(len>>3, GFP_ATOMIC);
> > >   	if (!new)
> > >   		return 0;
> > >
> > > -	bitmap_copy(new, map->tsn_map, map->max_tsn_seen - map->base_tsn);
> > > +	bitmap_copy(new, map->tsn_map, map->max_tsn_seen - map->base_tsn + 1);
> >
> > Can simplify that this by using map->cumulative_tsn_ack_point instead of
> > base_tsn.
> >
> > -vlad
> >
> 
> I changed the code to use "cumulative_tsn_ack_point" in an updated version
> of the patch.
> 
>                                          -- Lee
> 
> > >   	kfree(map->tsn_map);
> > >   	map->tsn_map = new;
> > >   	map->len = len;
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-sctp" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >

[PATCH 0/4] sctp: fix association hangs due to reassembly/ordering logic

[Lee A. Roberts]

From: "Lee A. Roberts" <lee.roberts@hp.com>

This series of patches resolves several SCTP association hangs observed during
SCTP stress testing.  Observable symptoms include communications hangs with
data being held in the association reassembly and/or lobby (ordering) queues.
Close examination of reassembly/ordering queues may show either duplicated
or missing packets.

Lee A. Roberts (4):
  sctp: fix association hangs due to off-by-one errors in
    sctp_tsnmap_grow()
  sctp: fix association hangs due to reneging packets below the
    cumulative TSN ACK point
  sctp: fix association hangs due to errors when reneging events from
    the ordering queue
  sctp: fix association hangs due to partial delivery errors

 net/sctp/tsnmap.c   |   13 ++++----
 net/sctp/ulpqueue.c |   87 +++++++++++++++++++++++++++++++++++++++++----------
 2 files changed, 78 insertions(+), 22 deletions(-)

-- 
1.7.9.5

[PATCH 1/4] sctp: fix association hangs due to off-by-one errors in sctp_tsnmap_grow()

[Lee A. Roberts]

From: "Lee A. Roberts" <lee.roberts@hp.com>

Resolve SCTP association hangs observed during SCTP stress
testing.  Observable symptoms include communications hangs
with data being held in the association lobby (ordering)
queue.  Close examination of reassembly/ordering queues shows
duplicated packets.

In sctp_tsnmap_mark(), correct off-by-one error when calculating
size value for sctp_tsnmap_grow().

In sctp_tsnmap_grow(), correct off-by-one error when copying
and resizing the tsnmap.  If max_tsn_seen is in the LSB of the
word, this bit can be lost, causing the corresponding packet
to be transmitted again and to be entered as a duplicate into
the SCTP reassembly/ordering queues.  Change parameter name
from "gap" (zero-based index) to "size" (one-based) to enhance
code readability.

Signed-off-by: Lee A. Roberts <lee.roberts@hp.com>
---
 net/sctp/tsnmap.c |   13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/net/sctp/tsnmap.c b/net/sctp/tsnmap.c
index 5f25e0c..396c451 100644
--- a/net/sctp/tsnmap.c
+++ b/net/sctp/tsnmap.c
@@ -51,7 +51,7 @@
 static void sctp_tsnmap_update(struct sctp_tsnmap *map);
 static void sctp_tsnmap_find_gap_ack(unsigned long *map, __u16 off,
 				     __u16 len, __u16 *start, __u16 *end);
-static int sctp_tsnmap_grow(struct sctp_tsnmap *map, u16 gap);
+static int sctp_tsnmap_grow(struct sctp_tsnmap *map, u16 size);
 
 /* Initialize a block of memory as a tsnmap.  */
 struct sctp_tsnmap *sctp_tsnmap_init(struct sctp_tsnmap *map, __u16 len,
@@ -124,7 +124,7 @@ int sctp_tsnmap_mark(struct sctp_tsnmap *map, __u32 tsn,
 
 	gap = tsn - map->base_tsn;
 
-	if (gap >= map->len && !sctp_tsnmap_grow(map, gap))
+	if (gap >= map->len && !sctp_tsnmap_grow(map, gap + 1))
 		return -ENOMEM;
 
 	if (!sctp_tsnmap_has_gap(map) && gap == 0) {
@@ -360,23 +360,24 @@ __u16 sctp_tsnmap_num_gabs(struct sctp_tsnmap *map,
 	return ngaps;
 }
 
-static int sctp_tsnmap_grow(struct sctp_tsnmap *map, u16 gap)
+static int sctp_tsnmap_grow(struct sctp_tsnmap *map, u16 size)
 {
 	unsigned long *new;
 	unsigned long inc;
 	u16  len;
 
-	if (gap >= SCTP_TSN_MAP_SIZE)
+	if (size > SCTP_TSN_MAP_SIZE)
 		return 0;
 
-	inc = ALIGN((gap - map->len),BITS_PER_LONG) + SCTP_TSN_MAP_INCREMENT;
+	inc = ALIGN((size - map->len), BITS_PER_LONG) + SCTP_TSN_MAP_INCREMENT;
 	len = min_t(u16, map->len + inc, SCTP_TSN_MAP_SIZE);
 
 	new = kzalloc(len>>3, GFP_ATOMIC);
 	if (!new)
 		return 0;
 
-	bitmap_copy(new, map->tsn_map, map->max_tsn_seen - map->base_tsn);
+	bitmap_copy(new, map->tsn_map,
+		map->max_tsn_seen - map->cumulative_tsn_ack_point);
 	kfree(map->tsn_map);
 	map->tsn_map = new;
 	map->len = len;
-- 
1.7.9.5

Re: [PATCH 1/4] sctp: fix association hangs due to off-by-one errors in sctp_tsnmap_grow()

[Vlad Yasevich]

On 02/26/2013 09:36 AM, Lee A. Roberts wrote:
> From: "Lee A. Roberts" <lee.roberts@hp.com>
>
> Resolve SCTP association hangs observed during SCTP stress
> testing.  Observable symptoms include communications hangs
> with data being held in the association lobby (ordering)
> queue.  Close examination of reassembly/ordering queues shows
> duplicated packets.
>
> In sctp_tsnmap_mark(), correct off-by-one error when calculating
> size value for sctp_tsnmap_grow().
>
> In sctp_tsnmap_grow(), correct off-by-one error when copying
> and resizing the tsnmap.  If max_tsn_seen is in the LSB of the
> word, this bit can be lost, causing the corresponding packet
> to be transmitted again and to be entered as a duplicate into
> the SCTP reassembly/ordering queues.  Change parameter name
> from "gap" (zero-based index) to "size" (one-based) to enhance
> code readability.
>
> Signed-off-by: Lee A. Roberts <lee.roberts@hp.com>

Acked-by: Vlad Yasevich <vyasevich@gmail.com>

-vlad

> ---
>   net/sctp/tsnmap.c |   13 +++++++------
>   1 file changed, 7 insertions(+), 6 deletions(-)
>
> diff --git a/net/sctp/tsnmap.c b/net/sctp/tsnmap.c
> index 5f25e0c..396c451 100644
> --- a/net/sctp/tsnmap.c
> +++ b/net/sctp/tsnmap.c
> @@ -51,7 +51,7 @@
>   static void sctp_tsnmap_update(struct sctp_tsnmap *map);
>   static void sctp_tsnmap_find_gap_ack(unsigned long *map, __u16 off,
>   				     __u16 len, __u16 *start, __u16 *end);
> -static int sctp_tsnmap_grow(struct sctp_tsnmap *map, u16 gap);
> +static int sctp_tsnmap_grow(struct sctp_tsnmap *map, u16 size);
>
>   /* Initialize a block of memory as a tsnmap.  */
>   struct sctp_tsnmap *sctp_tsnmap_init(struct sctp_tsnmap *map, __u16 len,
> @@ -124,7 +124,7 @@ int sctp_tsnmap_mark(struct sctp_tsnmap *map, __u32 tsn,
>
>   	gap = tsn - map->base_tsn;
>
> -	if (gap >= map->len && !sctp_tsnmap_grow(map, gap))
> +	if (gap >= map->len && !sctp_tsnmap_grow(map, gap + 1))
>   		return -ENOMEM;
>
>   	if (!sctp_tsnmap_has_gap(map) && gap == 0) {
> @@ -360,23 +360,24 @@ __u16 sctp_tsnmap_num_gabs(struct sctp_tsnmap *map,
>   	return ngaps;
>   }
>
> -static int sctp_tsnmap_grow(struct sctp_tsnmap *map, u16 gap)
> +static int sctp_tsnmap_grow(struct sctp_tsnmap *map, u16 size)
>   {
>   	unsigned long *new;
>   	unsigned long inc;
>   	u16  len;
>
> -	if (gap >= SCTP_TSN_MAP_SIZE)
> +	if (size > SCTP_TSN_MAP_SIZE)
>   		return 0;
>
> -	inc = ALIGN((gap - map->len),BITS_PER_LONG) + SCTP_TSN_MAP_INCREMENT;
> +	inc = ALIGN((size - map->len), BITS_PER_LONG) + SCTP_TSN_MAP_INCREMENT;
>   	len = min_t(u16, map->len + inc, SCTP_TSN_MAP_SIZE);
>
>   	new = kzalloc(len>>3, GFP_ATOMIC);
>   	if (!new)
>   		return 0;
>
> -	bitmap_copy(new, map->tsn_map, map->max_tsn_seen - map->base_tsn);
> +	bitmap_copy(new, map->tsn_map,
> +		map->max_tsn_seen - map->cumulative_tsn_ack_point);
>   	kfree(map->tsn_map);
>   	map->tsn_map = new;
>   	map->len = len;
>