[PATCH v3] zstd: fixed possible 'rtbTable' underflow in FSE_normalizeCount()

[PATCH v3] zstd: fixed possible 'rtbTable' underflow in FSE_normalizeCount()

[Ilya Krutskih]

'rtbTable' may be underflowed because 'proba' is used without
checking for a non-negative as index of rtbTable[].

Add check: proba >= 0

Cc: stable@vger.kernel.org # v5.10+
Fixes: e0c1b49f5b67 ("lib: zstd: Upgrade to latest upstream zstd version 1.4.10")
Signed-off-by: Ilya Krutskih <devsec@tpz.ru>
---
 lib/zstd/compress/fse_compress.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/lib/zstd/compress/fse_compress.c b/lib/zstd/compress/fse_compress.c
index 44a3c10becf2..6b83f8bc943a 100644
--- a/lib/zstd/compress/fse_compress.c
+++ b/lib/zstd/compress/fse_compress.c
@@ -492,9 +492,10 @@ size_t FSE_normalizeCount (short* normalizedCounter, unsigned tableLog,
                 stillToDistribute--;
             } else {
                 short proba = (short)((count[s]*step) >> scale);
-                if (proba<8) {
-                    U64 restToBeat = vStep * rtbTable[proba];
-                    proba += (count[s]*step) - ((U64)proba<<scale) > restToBeat;
+		if ((proba >= 0) && (proba < 8)) {
+			U64 restToBeat = vStep * rtbTable[proba];
+
+			proba += (count[s]*step) - ((U64)proba<<scale) > restToBeat;
                 }
                 if (proba > largestP) { largestP=proba; largest=s; }
                 normalizedCounter[s] = proba;
-- 
2.43.0

Re: [PATCH v3] zstd: fixed possible 'rtbTable' underflow in FSE_normalizeCount()

[Felix Handte]

Ilya, can you share any context for this patch? Do you have any evidence 
that `proba` can be negative?

A discussion was just started about this patch on the zstd repo [0]. I'm 
happy to discuss this here or there, whichever is more convenient.

But to my first pass inspection, this seems to be protecting an 
impossible situation. (Separately: if it could happen, the correct 
behavior would to catch it and return an error, not just skip it like 
this patch proposes.)

Thanks,
Felix

[0] https://github.com/facebook/zstd/issues/4567

On 12/11/25 12:19 PM, Ilya Krutskih wrote:
> 'rtbTable' may be underflowed because 'proba' is used without
> checking for a non-negative as index of rtbTable[].
> 
> Add check: proba >= 0
> 
> Cc: stable@vger.kernel.org # v5.10+
> Fixes: e0c1b49f5b67 ("lib: zstd: Upgrade to latest upstream zstd version 1.4.10")
> Signed-off-by: Ilya Krutskih <devsec@tpz.ru>
> ---
>   lib/zstd/compress/fse_compress.c | 7 ++++---
>   1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/lib/zstd/compress/fse_compress.c b/lib/zstd/compress/fse_compress.c
> index 44a3c10becf2..6b83f8bc943a 100644
> --- a/lib/zstd/compress/fse_compress.c
> +++ b/lib/zstd/compress/fse_compress.c
> @@ -492,9 +492,10 @@ size_t FSE_normalizeCount (short* normalizedCounter, unsigned tableLog,
>                   stillToDistribute--;
>               } else {
>                   short proba = (short)((count[s]*step) >> scale);
> -                if (proba<8) {
> -                    U64 restToBeat = vStep * rtbTable[proba];
> -                    proba += (count[s]*step) - ((U64)proba<<scale) > restToBeat;
> +		if ((proba >= 0) && (proba < 8)) {
> +			U64 restToBeat = vStep * rtbTable[proba];
> +
> +			proba += (count[s]*step) - ((U64)proba<<scale) > restToBeat;
>                   }
>                   if (proba > largestP) { largestP=proba; largest=s; }
>                   normalizedCounter[s] = proba;