[DPDK/cryptodev Bug 2126] drivers/crypto/openssl/rte_openssl_pmd.c:2268 in process_openssl_modexp_op()

[DPDK/cryptodev Bug 2126] drivers/crypto/openssl/rte_openssl_pmd.c:2268 in process_openssl_modexp_op()

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=2126

            Bug ID: 2126
           Summary: drivers/crypto/openssl/rte_openssl_pmd.c:2268  in
                    process_openssl_modexp_op()
           Product: DPDK
           Version: 26.11
          Hardware: All
                OS: All
            Status: UNCONFIRMED
          Severity: normal
          Priority: Normal
         Component: cryptodev
          Assignee: dev@dpdk.org
          Reporter: stephen@networkplumber.org
  Target Milestone: ---

Reported-by Anthropic OSS Scanner ANT-2026-71V1RCLL

Every modexp and modinv op processed by the `crypto_openssl` PMD permanently
adds two BIGNUMs to the session's `BN_CTX`. A long-lived session therefore
grows the process heap by a fixed amount per op (1124 bytes per 2048-bit
modexp, 628 per modinv) until the session is freed. An application that keeps
one of these sessions and runs many ops on it will eventually see ops complete
with `RTE_CRYPTO_OP_STATUS_ERROR` once allocation fails inside `BN_mod_exp()`,
or the kernel kills the process if no memory limit is set.

The cause is in `process_openssl_modexp_op()` and `process_openssl_modinv_op()`
in `drivers/crypto/openssl/rte_openssl_pmd.c`. Each calls `BN_CTX_get()` twice
on the session context without its own `BN_CTX_start()`/`BN_CTX_end()` pair.
The only open frame on that context is the one
`openssl_set_asym_session_parameters()` opens at session creation, and
`openssl_reset_asym_session()` closes it at session release, so every op takes
two new entries from the pool and none are ever reused. `BN_clear()` only
zeroes the values. The default build is affected, no special input is involved,
and ASan/LSan report nothing because the memory is released with the session.
The attached reproducer runs 2 × 10000 modexp ops on one session and shows
about 11 MB of growth per batch, nearly all of it returned by
`rte_cryptodev_asym_session_free()`. `dpdk-test-crypto-perf --optype modex
--total-ops 1000000` peaks around 631 MiB RSS for the same reason. The attached
patch wraps each op in its own nested `BN_CTX_start()`/`BN_CTX_end()` frame,
which brings per-op growth to zero. It would carry `Fixes: 3e9d6bd447fb` and
`Cc: stable`, like the earlier per-op leak fixes in this driver.

This is an ordinary bug, not a security issue: the ops that trigger it are
well-formed, nothing in DPDK lets an
outside party drive modexp or modinv ops on a `crypto_openssl` session, and an
application's own ordinary load hits
it just the same.

#

-- 
You are receiving this mail because:
You are the assignee for the bug.