topology: harden binary topology decoder against malformed input

[GitHub pull_request - opened <github@alsa-project.org>]

alsa-project/alsa-lib pull request #526 was opened from HarshRajSinghania:

As discussed over email with Jaroslav, this hardens the binary topology decoder
against malformed / untrusted `.tplg` input. Three related issues, each confirmed
with AddressSanitizer:

1. `snd_tplg_decode()` (src/topology/decoder.c): `hdr->size + hdr->payload_size`
   is evaluated in 32-bit and can wrap, bypassing the payload bounds check; the
   block decoder is then called with the raw (huge) `payload_size`, causing an
   out-of-bounds read. Fixed by doing the arithmetic in 64-bit (and the advance
   in `size_t`).

2. `tplg_decode_dapm_widget()` (src/topology/dapm.c): the kcontrol loop reads the
   control header (`chdr->type`, then `mc/ec/bc->size` and `->priv.size`) before
   the `size2 > size` check, giving an out-of-bounds read on a truncated payload.
   Fixed by adding `size < sizeof(*chdr)` / `sizeof(*mc|ec|bc)` guards and
   computing `size2` in `size_t`.

3. `tplg_decode_dapm_graph()` (src/topology/dapm.c): `alloca()` is called with a
   size derived from the input payload, allowing stack exhaustion on large input.
   Fixed by switching to `calloc()` / `free()`, which fails gracefully.

Scope is the debug-oriented decode path only, as you noted. Standalone ASan PoC
harnesses are available if useful.

Signed-off-by: Harsh Raj Singhania <raj.harshraut@gmail.com>

Request URL   : https://github.com/alsa-project/alsa-lib/pull/526
Patch URL     : https://github.com/alsa-project/alsa-lib/pull/526.patch
Repository URL: https://github.com/alsa-project/alsa-lib