Re: [PATCH net v2] net/smc: fix application data exception

[Heiko Carstens <hca@linux.ibm.com>]

On Thu, Feb 16, 2023 at 02:39:05PM +0800, D. Wythe wrote:
> From: "D. Wythe" <alibuda@linux.alibaba.com>
> 
> There is a certain probability that following
> exceptions will occur in the wrk benchmark test:
> 
> Running 10s test @ http://11.213.45.6:80
>   8 threads and 64 connections
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency     3.72ms   13.94ms 245.33ms   94.17%
>     Req/Sec     1.96k   713.67     5.41k    75.16%
>   155262 requests in 10.10s, 23.10MB read
> Non-2xx or 3xx responses: 3
> 
> We will find that the error is HTTP 400 error, which is a serious
> exception in our test, which means the application data was
> corrupted.
> 
> Consider the following scenarios:
> 
> CPU0                            CPU1
> 
> buf_desc->used = 0;
>                                 cmpxchg(buf_desc->used, 0, 1)
>                                 deal_with(buf_desc)
> 
> memset(buf_desc->cpu_addr,0);
> 
> This will cause the data received by a victim connection to be cleared,
> thus triggering an HTTP 400 error in the server.
> 
> This patch exchange the order between clear used and memset, add
> barrier to ensure memory consistency.
> 
> Fixes: 1c5526968e27 ("net/smc: Clear memory when release and reuse buffer")
> Signed-off-by: D. Wythe <alibuda@linux.alibaba.com>
> ---
> v2: rebase it with latest net tree.
> 
>  net/smc/smc_core.c | 17 ++++++++---------
>  1 file changed, 8 insertions(+), 9 deletions(-)
> 
> diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c
> index c305d8d..c19d4b7 100644
> --- a/net/smc/smc_core.c
> +++ b/net/smc/smc_core.c
> @@ -1120,8 +1120,9 @@ static void smcr_buf_unuse(struct smc_buf_desc *buf_desc, bool is_rmb,
>  
>  		smc_buf_free(lgr, is_rmb, buf_desc);
>  	} else {
> -		buf_desc->used = 0;
> -		memset(buf_desc->cpu_addr, 0, buf_desc->len);
> +		/* memzero_explicit provides potential memory barrier semantics */
> +		memzero_explicit(buf_desc->cpu_addr, buf_desc->len);
> +		WRITE_ONCE(buf_desc->used, 0);

This looks odd to me. memzero_explicit() is only sort of a compiler
barrier, since it is a function call, but not a real memory barrier.

You may want to check Documentation/memory-barriers.txt and
Documentation/atomic_t.txt.

To me the proper solution looks like buf_desc->used should be converted to
an atomic_t, and then you could do:

	memset(buf_desc->cpu_addr, 0, buf_desc->len);
	smp_mb__before_atomic();
	atomic_set(&buf_desc->used, 0);

and in a similar way use atomic_cmpxchg() instead of the now used cmpxchg()
for the part that sets buf_desc->used to 1.

Adding experts to cc, since s390 has such strong memory ordering semantics
that you can basically do whatever you want without breaking anything. So I
don't consider myself an expert here at all. :)

But given that this is common code, let's make sure this is really correct.