ksmbd: refactor security descriptor handling?

[Ian Bridges <icb@fastmail.org>]

Hi Namjae,

Would you be open to a refactor of the security descriptor (SD)
handling in fs/smb/server/smbacl.c?

Since February 2025 smbacl.c has taken 24 fixes for bounds checking,
sizing and SID validation, 20 of them this year. Several readers walk
the SD bytes independently, each with its own validation
(parse_sec_desc() and parse_dacl(), smb_inherit_dacl(),
smb_check_perm_dacl() and build_sec_desc()), and the QUERY_INFO reply
is sized by an estimate and then written ACE by ACE.

Roughly, the plan would be
 - One decoder, one encoder and one decoded type. The decoder checks
   the whole SD once and returns a structure that callers read and
   build from, without handling SD bytes themselves. The encoder
   computes the exact size first and bounds every write by it.
 - QUERY_INFO, SET_INFO and inheritance moved onto them one caller at
   a time. The access check stays as it is.
 - Current behaviour kept, apart from rejecting a malformed SD before
   it is stored or used. Everything stays inside fs/smb/server.

I would test it with KUnit tests next to the existing smbacl tests,
smbtorture's smb2.acls tests, and by comparing each converted path
against the unmodified module. I would also check the decoder and the
encoder with CBMC, a bounded model checker, for memory safety on every
SD up to a fixed size, and can share the harnesses.

If you are open to it, I would send a prototype as an RFC series. Is
anyone already working in this area, or is there anything you would
like done differently?

Thanks,
Ian