* [PATCH 0/2] module: Tweak return and warning
From: Lucas De Marchi @ 2025-10-13 16:26 UTC (permalink / raw)
To: linux-modules, linux-kernel; +Cc: Lucas De Marchi, Petr Pavlu
Do not let userspace tools and end users confused on the return code and
log messages.
Signed-off-by: Lucas De Marchi <lucas.demarchi@intel.com>
---
Lucas De Marchi (2):
module: Override -EEXISTS module return
module: Simplify warning on positive returns from module_init()
kernel/module/main.c | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
base-commit: e5f0a698b34ed76002dc5cff3804a61c80233a7a
change-id: 20251013-module-warn-ret-59f085298055
Lucas De Marchi
^ permalink raw reply
* [PATCH 2/2] module: Simplify warning on positive returns from module_init()
From: Lucas De Marchi @ 2025-10-13 16:26 UTC (permalink / raw)
To: linux-modules, linux-kernel; +Cc: Lucas De Marchi, Petr Pavlu
In-Reply-To: <20251013-module-warn-ret-v1-0-ab65b41af01f@intel.com>
It should now be rare to trigger this warning - it doesn't need to be so
verbose. Make it follow the usual style in the module loading code.
For the same reason, drop the dump_stack().
Suggested-by: Petr Pavlu <petr.pavlu@suse.com>
Signed-off-by: Lucas De Marchi <lucas.demarchi@intel.com>
---
kernel/module/main.c | 10 +++-------
1 file changed, 3 insertions(+), 7 deletions(-)
diff --git a/kernel/module/main.c b/kernel/module/main.c
index 74ff87b13c517..31c54bf6df4b2 100644
--- a/kernel/module/main.c
+++ b/kernel/module/main.c
@@ -3045,13 +3045,9 @@ static noinline int do_init_module(struct module *mod)
}
goto fail_free_freeinit;
}
- if (ret > 0) {
- pr_warn("%s: '%s'->init suspiciously returned %d, it should "
- "follow 0/-E convention\n"
- "%s: loading module anyway...\n",
- __func__, mod->name, ret, __func__);
- dump_stack();
- }
+ if (ret > 0)
+ pr_warn("%s: init suspiciously returned %d, it should follow 0/-E convention\n",
+ mod->name, ret);
/* Now it's a first class citizen! */
mod->state = MODULE_STATE_LIVE;
--
2.51.0
^ permalink raw reply related
* [PATCH 1/2] module: Override -EEXISTS module return
From: Lucas De Marchi @ 2025-10-13 16:26 UTC (permalink / raw)
To: linux-modules, linux-kernel; +Cc: Lucas De Marchi, Petr Pavlu
In-Reply-To: <20251013-module-warn-ret-v1-0-ab65b41af01f@intel.com>
The -EEXIST errno is reserved by the module loading functionality. When
userspace calls [f]init_module(), it expects a -EEXIST to mean that the
module is already loaded in the kernel. If module_init() returns it,
that is not true anymore.
Add a warning and override the return code to workaround modules
currently returning the wrong code. It's expected that they eventually
migrate to a better suited error.
Closes: https://lore.kernel.org/all/aKLzsAX14ybEjHfJ@orbyte.nwl.cc/
Signed-off-by: Lucas De Marchi <lucas.demarchi@intel.com>
---
kernel/module/main.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/kernel/module/main.c b/kernel/module/main.c
index c66b261849362..74ff87b13c517 100644
--- a/kernel/module/main.c
+++ b/kernel/module/main.c
@@ -3038,6 +3038,11 @@ static noinline int do_init_module(struct module *mod)
if (mod->init != NULL)
ret = do_one_initcall(mod->init);
if (ret < 0) {
+ if (ret == -EEXIST) {
+ pr_warn("%s: init suspiciously returned -EEXIST: Overriding with -EBUSY\n",
+ mod->name);
+ ret = -EBUSY;
+ }
goto fail_free_freeinit;
}
if (ret > 0) {
--
2.51.0
^ permalink raw reply related
* Re: [PATCH v2 00/10] scalable symbol flags with __kflagstab
From: Jonathan Corbet @ 2025-10-13 19:02 UTC (permalink / raw)
To: Siddharth Nayyar, petr.pavlu
Cc: arnd, linux-arch, linux-kbuild, linux-kernel, linux-modules,
mcgrof, nathan, nicolas.schier, samitolvanen, sidnayyar, maennich,
gprocida
In-Reply-To: <20251013153918.2206045-1-sidnayyar@google.com>
Siddharth Nayyar <sidnayyar@google.com> writes:
> This patch series implements a mechanism for scalable exported symbol
> flags using a separate section called __kflagstab. The series introduces
> __kflagstab support, removes *_gpl sections in favor of a GPL flag,
> simplifies symbol resolution during module loading, and adds symbol
> import protection.
This caught my eye in passing ... some questions ...
The import protection would appear to be the real point of this work?
But it seems that you have kind of buried it; why not describe what you
are trying to do here and how it will be used?
I ask "how it will be used" since you don't provide any way to actually
mark exports with this new flag. What is the intended usage here?
If I understand things correctly, applying this series will immediately
result in the inability to load any previously built modules, right?
That will create a sort of flag day for anybody with out-of-tree modules
that some may well see as a regression. Is that really the intent?
Thanks,
jon
^ permalink raw reply
* Re: [PATCH v17 28/47] dept: add documentation for dept
From: NeilBrown @ 2025-10-14 6:03 UTC (permalink / raw)
To: Byungchul Park
Cc: linux-kernel, kernel_team, torvalds, damien.lemoal, linux-ide,
adilger.kernel, linux-ext4, mingo, peterz, will, tglx, rostedt,
joel, sashal, daniel.vetter, duyuyang, johannes.berg, tj, tytso,
willy, david, amir73il, gregkh, kernel-team, linux-mm, akpm,
mhocko, minchan, hannes, vdavydov.dev, sj, jglisse, dennis, cl,
penberg, rientjes, vbabka, ngupta, linux-block, josef,
linux-fsdevel, jack, jlayton, dan.j.williams, hch, djwong,
dri-devel, rodrigosiqueiramelo, melissa.srw, hamohammed.sa,
harry.yoo, chris.p.wilson, gwan-gyeong.mun, max.byungchul.park,
boqun.feng, longman, yunseong.kim, ysk, yeoreum.yun, netdev,
matthew.brost, her0gyugyu, corbet, catalin.marinas, bp,
dave.hansen, x86, hpa, luto, sumit.semwal, gustavo,
christian.koenig, andi.shyti, arnd, lorenzo.stoakes, Liam.Howlett,
rppt, surenb, mcgrof, petr.pavlu, da.gomez, samitolvanen, paulmck,
frederic, neeraj.upadhyay, joelagnelf, josh, urezki,
mathieu.desnoyers, jiangshanlai, qiang.zhang, juri.lelli,
vincent.guittot, dietmar.eggemann, bsegall, mgorman, vschneid,
chuck.lever, okorniev, Dai.Ngo, tom, trondmy, anna, kees, bigeasy,
clrkwllms, mark.rutland, ada.coupriediaz, kristina.martsenko,
wangkefeng.wang, broonie, kevin.brodsky, dwmw, shakeel.butt, ast,
ziy, yuzhao, baolin.wang, usamaarif642, joel.granados,
richard.weiyang, geert+renesas, tim.c.chen, linux,
alexander.shishkin, lillian, chenhuacai, francesco,
guoweikang.kernel, link, jpoimboe, masahiroy, brauner,
thomas.weissschuh, oleg, mjguzik, andrii, wangfushuai, linux-doc,
linux-arm-kernel, linux-media, linaro-mm-sig, linux-i2c,
linux-arch, linux-modules, rcu, linux-nfs, linux-rt-devel
In-Reply-To: <20251013052354.GA75512@system.software.com>
On Mon, 13 Oct 2025, Byungchul Park wrote:
> On Fri, Oct 03, 2025 at 04:55:14PM +1000, NeilBrown wrote:
> > On Thu, 02 Oct 2025, Byungchul Park wrote:
> > > This document describes the concept and APIs of dept.
> > >
> >
> > Thanks for the documentation. I've been trying to understand it.
>
> You're welcome. Feel free to ask me if you have any questions.
>
> > > +How DEPT works
> > > +--------------
> > > +
> > > +Let's take a look how DEPT works with the 1st example in the section
> > > +'Limitation of lockdep'.
> > > +
> > > + context X context Y context Z
> > > +
> > > + mutex_lock A
> > > + folio_lock B
> > > + folio_lock B <- DEADLOCK
> > > + mutex_lock A <- DEADLOCK
> > > + folio_unlock B
> > > + folio_unlock B
> > > + mutex_unlock A
> > > + mutex_unlock A
> > > +
> > > +Adding comments to describe DEPT's view in terms of wait and event:
> > > +
> > > + context X context Y context Z
> > > +
> > > + mutex_lock A
> > > + /* wait for A */
> > > + folio_lock B
> > > + /* wait for A */
> > > + /* start event A context */
> > > +
> > > + folio_lock B
> > > + /* wait for B */ <- DEADLOCK
> > > + /* start event B context */
> > > +
> > > + mutex_lock A
> > > + /* wait for A */ <- DEADLOCK
> > > + /* start event A context */
> > > +
> > > + folio_unlock B
> > > + /* event B */
> > > + folio_unlock B
> > > + /* event B */
> > > +
> > > + mutex_unlock A
> > > + /* event A */
> > > + mutex_unlock A
> > > + /* event A */
> > > +
> >
> > I can't see the value of the above section.
> > The first section with no comments is useful as it is easy to see the
> > deadlock being investigate. The section below is useful as it add
> > comments to explain how DEPT sees the situation. But the above section,
> > with some but not all of the comments, does seem (to me) to add anything
> > useful.
>
> I just wanted to convert 'locking terms' to 'wait and event terms' by
> one step. However, I can remove the section you pointed out that you
> thought was useless.
But it seems you did it in two steps???
If you think the middle section with some but not all of the comments
adds value (And maybe it does - maybe I just haven't seen it yet), the
please explain what value is being added at each step.
It is currently documented as:
+Adding comments to describe DEPT's view in terms of wait and event:
then
+Adding more supplementary comments to describe DEPT's view in detail:
Maybe if you said more DEPT's view so at this point so that when we see
the supplementary comments, we can understand how they relate to DEPT's
view.
>
> > > +
> > > + context X context Y context Z
> > > +
> > > + mutex_lock A
> > > + /* might wait for A */
> > > + /* start to take into account event A's context */
> >
> > What do you mean precisely by "context".
>
> That means one of task context, irq context, wq worker context (even
> though it can also be considered as task context), or something.
OK, that makes sense. If you provide this definition for "context"
before you use the term, I think that will help the reader.
> > If the examples that follow It seems that the "context" for event A
> > starts at "mutex lock A" when it (possibly) waits for a mutex and ends
> > at "mutex unlock A" - which are both in the same process. Clearly
> > various other events that happen between these two points in the same
> > process could be seen as the "context" for event A.
> >
> > However event B starts in "context X" with "folio_lock B" and ends in
> > "context Z" or "context Y" with "folio_unlock B". Is that right?
>
> Right.
>
> > My question then is: how do you decide which, of all the event in all
> > the processes in all the system, between the start[S] and the end[E] are
> > considered to be part of the "context" of event A.
>
> DEPT can identify the "context" of event A only *once* the event A is
> actually executed, and builds dependencies between the event and the
> recorded waits in the "context" of event A since [S].
So a dependency is an ordered set of pairs of "context" and "wait" or
"context" and "event". "wait"s and "event"s are linked by some abstract
identifier for the event (like lockdep's lock classes).
How are the contexts abstracted. Is it just "same" or "different"
I'll try reading the document again and see how much further I get.
Thanks,
NeilBrown
^ permalink raw reply
* Re: [PATCH v17 28/47] dept: add documentation for dept
From: Byungchul Park @ 2025-10-14 6:38 UTC (permalink / raw)
To: NeilBrown
Cc: linux-kernel, kernel_team, torvalds, damien.lemoal, linux-ide,
adilger.kernel, linux-ext4, mingo, peterz, will, tglx, rostedt,
joel, sashal, daniel.vetter, duyuyang, johannes.berg, tj, tytso,
willy, david, amir73il, gregkh, kernel-team, linux-mm, akpm,
mhocko, minchan, hannes, vdavydov.dev, sj, jglisse, dennis, cl,
penberg, rientjes, vbabka, ngupta, linux-block, josef,
linux-fsdevel, jack, jlayton, dan.j.williams, hch, djwong,
dri-devel, rodrigosiqueiramelo, melissa.srw, hamohammed.sa,
harry.yoo, chris.p.wilson, gwan-gyeong.mun, max.byungchul.park,
boqun.feng, longman, yunseong.kim, ysk, yeoreum.yun, netdev,
matthew.brost, her0gyugyu, corbet, catalin.marinas, bp,
dave.hansen, x86, hpa, luto, sumit.semwal, gustavo,
christian.koenig, andi.shyti, arnd, lorenzo.stoakes, Liam.Howlett,
rppt, surenb, mcgrof, petr.pavlu, da.gomez, samitolvanen, paulmck,
frederic, neeraj.upadhyay, joelagnelf, josh, urezki,
mathieu.desnoyers, jiangshanlai, qiang.zhang, juri.lelli,
vincent.guittot, dietmar.eggemann, bsegall, mgorman, vschneid,
chuck.lever, okorniev, Dai.Ngo, tom, trondmy, anna, kees, bigeasy,
clrkwllms, mark.rutland, ada.coupriediaz, kristina.martsenko,
wangkefeng.wang, broonie, kevin.brodsky, dwmw, shakeel.butt, ast,
ziy, yuzhao, baolin.wang, usamaarif642, joel.granados,
richard.weiyang, geert+renesas, tim.c.chen, linux,
alexander.shishkin, lillian, chenhuacai, francesco,
guoweikang.kernel, link, jpoimboe, masahiroy, brauner,
thomas.weissschuh, oleg, mjguzik, andrii, wangfushuai, linux-doc,
linux-arm-kernel, linux-media, linaro-mm-sig, linux-i2c,
linux-arch, linux-modules, rcu, linux-nfs, linux-rt-devel
In-Reply-To: <176042183810.1793333.13639772065939276568@noble.neil.brown.name>
On Tue, Oct 14, 2025 at 05:03:58PM +1100, NeilBrown wrote:
> On Mon, 13 Oct 2025, Byungchul Park wrote:
> > On Fri, Oct 03, 2025 at 04:55:14PM +1000, NeilBrown wrote:
> > > On Thu, 02 Oct 2025, Byungchul Park wrote:
> > > > This document describes the concept and APIs of dept.
> > > >
> > >
> > > Thanks for the documentation. I've been trying to understand it.
> >
> > You're welcome. Feel free to ask me if you have any questions.
> >
> > > > +How DEPT works
> > > > +--------------
> > > > +
> > > > +Let's take a look how DEPT works with the 1st example in the section
> > > > +'Limitation of lockdep'.
> > > > +
> > > > + context X context Y context Z
> > > > +
> > > > + mutex_lock A
> > > > + folio_lock B
> > > > + folio_lock B <- DEADLOCK
> > > > + mutex_lock A <- DEADLOCK
> > > > + folio_unlock B
> > > > + folio_unlock B
> > > > + mutex_unlock A
> > > > + mutex_unlock A
> > > > +
> > > > +Adding comments to describe DEPT's view in terms of wait and event:
> > > > +
> > > > + context X context Y context Z
> > > > +
> > > > + mutex_lock A
> > > > + /* wait for A */
> > > > + folio_lock B
> > > > + /* wait for A */
> > > > + /* start event A context */
> > > > +
> > > > + folio_lock B
> > > > + /* wait for B */ <- DEADLOCK
> > > > + /* start event B context */
> > > > +
> > > > + mutex_lock A
> > > > + /* wait for A */ <- DEADLOCK
> > > > + /* start event A context */
> > > > +
> > > > + folio_unlock B
> > > > + /* event B */
> > > > + folio_unlock B
> > > > + /* event B */
> > > > +
> > > > + mutex_unlock A
> > > > + /* event A */
> > > > + mutex_unlock A
> > > > + /* event A */
> > > > +
> > >
> > > I can't see the value of the above section.
> > > The first section with no comments is useful as it is easy to see the
> > > deadlock being investigate. The section below is useful as it add
> > > comments to explain how DEPT sees the situation. But the above section,
> > > with some but not all of the comments, does seem (to me) to add anything
> > > useful.
> >
> > I just wanted to convert 'locking terms' to 'wait and event terms' by
> > one step. However, I can remove the section you pointed out that you
> > thought was useless.
>
> But it seems you did it in two steps???
>
> If you think the middle section with some but not all of the comments
> adds value (And maybe it does - maybe I just haven't seen it yet), the
> please explain what value is being added at each step.
>
> It is currently documented as:
>
> +Adding comments to describe DEPT's view in terms of wait and event:
>
> then
>
> +Adding more supplementary comments to describe DEPT's view in detail:
>
> Maybe if you said more DEPT's view so at this point so that when we see
> the supplementary comments, we can understand how they relate to DEPT's
> view.
As you pointed out, I'd better remove the middle part so as to simplify
it. It doesn't give much information I also think.
> > > > +
> > > > + context X context Y context Z
> > > > +
> > > > + mutex_lock A
> > > > + /* might wait for A */
> > > > + /* start to take into account event A's context */
> > >
> > > What do you mean precisely by "context".
> >
> > That means one of task context, irq context, wq worker context (even
> > though it can also be considered as task context), or something.
>
> OK, that makes sense. If you provide this definition for "context"
> before you use the term, I think that will help the reader.
Thank you. I will add it.
> > > If the examples that follow It seems that the "context" for event A
> > > starts at "mutex lock A" when it (possibly) waits for a mutex and ends
> > > at "mutex unlock A" - which are both in the same process. Clearly
> > > various other events that happen between these two points in the same
> > > process could be seen as the "context" for event A.
> > >
> > > However event B starts in "context X" with "folio_lock B" and ends in
> > > "context Z" or "context Y" with "folio_unlock B". Is that right?
> >
> > Right.
> >
> > > My question then is: how do you decide which, of all the event in all
> > > the processes in all the system, between the start[S] and the end[E] are
> > > considered to be part of the "context" of event A.
> >
> > DEPT can identify the "context" of event A only *once* the event A is
> > actually executed, and builds dependencies between the event and the
> > recorded waits in the "context" of event A since [S].
>
> So a dependency is an ordered set of pairs of "context" and "wait" or
I don't get what you were trying to tell here. FWIW, DEPT focuses on
*event* contexts and, within each event context, it tracks pairs of
waits that appears since [S] and the interesting event that identifies
the event context.
> "context" and "event". "wait"s and "event"s are linked by some abstract
> identifier for the event (like lockdep's lock classes).
Yeah, kind of.
> How are the contexts abstracted. Is it just "same" or "different"
I don't get this. Can you explain in more detail?
Byungchul
> I'll try reading the document again and see how much further I get.
>
> Thanks,
> NeilBrown
^ permalink raw reply
* Re: [PATCH v2 10/10] module loader: enforce symbol import protection
From: kernel test robot @ 2025-10-14 7:34 UTC (permalink / raw)
To: Siddharth Nayyar, petr.pavlu
Cc: llvm, oe-kbuild-all, arnd, linux-arch, linux-kbuild, linux-kernel,
linux-modules, mcgrof, nathan, nicolas.schier, samitolvanen,
sidnayyar, maennich, gprocida
In-Reply-To: <20251013153918.2206045-11-sidnayyar@google.com>
Hi Siddharth,
kernel test robot noticed the following build errors:
[auto build test ERROR on arnd-asm-generic/master]
[also build test ERROR on soc/for-next linus/master v6.18-rc1 next-20251013]
[cannot apply to mcgrof/modules-next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Siddharth-Nayyar/define-kernel-symbol-flags/20251014-005305
base: https://git.kernel.org/pub/scm/linux/kernel/git/arnd/asm-generic.git master
patch link: https://lore.kernel.org/r/20251013153918.2206045-11-sidnayyar%40google.com
patch subject: [PATCH v2 10/10] module loader: enforce symbol import protection
config: x86_64-kexec (https://download.01.org/0day-ci/archive/20251014/202510141538.VZqnRzHh-lkp@intel.com/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251014/202510141538.VZqnRzHh-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202510141538.VZqnRzHh-lkp@intel.com/
All errors (new ones prefixed by >>):
>> kernel/module/main.c:1271:32: error: no member named 'sig_ok' in 'struct module'
1271 | if (fsa.is_protected && !mod->sig_ok) {
| ~~~ ^
1 error generated.
vim +1271 kernel/module/main.c
1228
1229 /* Resolve a symbol for this module. I.e. if we find one, record usage. */
1230 static const struct kernel_symbol *resolve_symbol(struct module *mod,
1231 const struct load_info *info,
1232 const char *name,
1233 char ownername[])
1234 {
1235 struct find_symbol_arg fsa = {
1236 .name = name,
1237 .gplok = !(mod->taints & (1 << TAINT_PROPRIETARY_MODULE)),
1238 .warn = true,
1239 };
1240 int err;
1241
1242 /*
1243 * The module_mutex should not be a heavily contended lock;
1244 * if we get the occasional sleep here, we'll go an extra iteration
1245 * in the wait_event_interruptible(), which is harmless.
1246 */
1247 sched_annotate_sleep();
1248 mutex_lock(&module_mutex);
1249 if (!find_symbol(&fsa))
1250 goto unlock;
1251
1252 if (fsa.license == GPL_ONLY)
1253 mod->using_gplonly_symbols = true;
1254
1255 if (!inherit_taint(mod, fsa.owner, name)) {
1256 fsa.sym = NULL;
1257 goto getname;
1258 }
1259
1260 if (!check_version(info, name, mod, fsa.crc)) {
1261 fsa.sym = ERR_PTR(-EINVAL);
1262 goto getname;
1263 }
1264
1265 err = verify_namespace_is_imported(info, fsa.sym, mod);
1266 if (err) {
1267 fsa.sym = ERR_PTR(err);
1268 goto getname;
1269 }
1270
> 1271 if (fsa.is_protected && !mod->sig_ok) {
1272 pr_warn("%s: Cannot use protected symbol %s\n",
1273 mod->name, name);
1274 fsa.sym = ERR_PTR(-EACCES);
1275 goto getname;
1276 }
1277
1278 err = ref_module(mod, fsa.owner);
1279 if (err) {
1280 fsa.sym = ERR_PTR(err);
1281 goto getname;
1282 }
1283
1284 getname:
1285 /* We must make copy under the lock if we failed to get ref. */
1286 strscpy(ownername, module_name(fsa.owner), MODULE_NAME_LEN);
1287 unlock:
1288 mutex_unlock(&module_mutex);
1289 return fsa.sym;
1290 }
1291
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* [PATCH] docs: ABI: sysfs-module: list all taint flags
From: Randy Dunlap @ 2025-10-15 22:13 UTC (permalink / raw)
To: linux-kernel
Cc: Randy Dunlap, Jonathan Corbet, linux-doc, Greg Kroah-Hartman,
Luis Chamberlain, Petr Pavlu, Daniel Gomez, Sami Tolvanen,
linux-modules
The list of module taint flags has not been updated lately as the
taint flags list grows. Instead of trying to keep multiple lists
updated, just refer to the list of kernel taint flags since they are
the same.
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
---
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: linux-doc@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Luis Chamberlain <mcgrof@kernel.org>
Cc: Petr Pavlu <petr.pavlu@suse.com>
Cc: Daniel Gomez <da.gomez@kernel.org>
Cc: Sami Tolvanen <samitolvanen@google.com>
Cc: linux-modules@vger.kernel.org
---
Documentation/ABI/testing/sysfs-module | 10 ++--------
Documentation/admin-guide/tainted-kernels.rst | 2 ++
2 files changed, 4 insertions(+), 8 deletions(-)
--- linux-next-20251014.orig/Documentation/ABI/testing/sysfs-module
+++ linux-next-20251014/Documentation/ABI/testing/sysfs-module
@@ -52,14 +52,8 @@ What: /sys/module/*/taint
Date: Jan 2012
KernelVersion: 3.3
Contact: Kay Sievers <kay.sievers@vrfy.org>
-Description: Module taint flags:
- == =====================
- P proprietary module
- O out-of-tree module
- F force-loaded module
- C staging driver module
- E unsigned module
- == =====================
+Description: Module taint flags: same as the kernel taint flags.
+ See: :ref:`taint_flags` in Documentation/admin-guide/tainted-kernels.rst
What: /sys/module/grant_table/parameters/free_per_iteration
Date: July 2023
--- linux-next-20251014.orig/Documentation/admin-guide/tainted-kernels.rst
+++ linux-next-20251014/Documentation/admin-guide/tainted-kernels.rst
@@ -76,6 +76,8 @@ which bits are set::
$ for i in $(seq 18); do echo $(($i-1)) $(($(cat /proc/sys/kernel/tainted)>>($i-1)&1));done
+.. _taint_flags:
+
Table for decoding tainted state
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
^ permalink raw reply
* Re: [PATCH v17 01/47] llist: move llist_{head,node} definition to types.h
From: Byungchul Park @ 2025-10-16 0:38 UTC (permalink / raw)
To: Greg KH
Cc: linux-kernel, kernel_team, torvalds, damien.lemoal, linux-ide,
adilger.kernel, linux-ext4, mingo, peterz, will, tglx, rostedt,
joel, sashal, daniel.vetter, duyuyang, johannes.berg, tj, tytso,
willy, david, amir73il, kernel-team, linux-mm, akpm, mhocko,
minchan, hannes, vdavydov.dev, sj, jglisse, dennis, cl, penberg,
rientjes, vbabka, ngupta, linux-block, josef, linux-fsdevel, jack,
jlayton, dan.j.williams, hch, djwong, dri-devel,
rodrigosiqueiramelo, melissa.srw, hamohammed.sa, harry.yoo,
chris.p.wilson, gwan-gyeong.mun, max.byungchul.park, boqun.feng,
longman, yunseong.kim, ysk, yeoreum.yun, netdev, matthew.brost,
her0gyugyu, corbet, catalin.marinas, bp, dave.hansen, x86, hpa,
luto, sumit.semwal, gustavo, christian.koenig, andi.shyti, arnd,
lorenzo.stoakes, Liam.Howlett, rppt, surenb, mcgrof, petr.pavlu,
da.gomez, samitolvanen, paulmck, frederic, neeraj.upadhyay,
joelagnelf, josh, urezki, mathieu.desnoyers, jiangshanlai,
qiang.zhang, juri.lelli, vincent.guittot, dietmar.eggemann,
bsegall, mgorman, vschneid, chuck.lever, neil, okorniev, Dai.Ngo,
tom, trondmy, anna, kees, bigeasy, clrkwllms, mark.rutland,
ada.coupriediaz, kristina.martsenko, wangkefeng.wang, broonie,
kevin.brodsky, dwmw, shakeel.butt, ast, ziy, yuzhao, baolin.wang,
usamaarif642, joel.granados, richard.weiyang, geert+renesas,
tim.c.chen, linux, alexander.shishkin, lillian, chenhuacai,
francesco, guoweikang.kernel, link, jpoimboe, masahiroy, brauner,
thomas.weissschuh, oleg, mjguzik, andrii, wangfushuai, linux-doc,
linux-arm-kernel, linux-media, linaro-mm-sig, linux-i2c,
linux-arch, linux-modules, rcu, linux-nfs, linux-rt-devel
In-Reply-To: <2025100230-grafted-alias-22a2@gregkh>
On Thu, Oct 02, 2025 at 10:24:41AM +0200, Greg KH wrote:
> On Thu, Oct 02, 2025 at 05:12:01PM +0900, Byungchul Park wrote:
> > llist_head and llist_node can be used by some other header files. For
> > example, dept for tracking dependencies uses llist in its header. To
> > avoid header dependency, move them to types.h.
>
> If you need llist in your code, then include llist.h. Don't force all
Eventually, I found out another way to avoid the dependency issue.
Thanks anyway for the feedback.
Byungchul
> types.h users to do so as there is not a dependency in types.h for
> llist.h.
>
> This patch shouldn't be needed as you are hiding "header dependency" for
> other files.
>
> thanks,
>
> greg k-h
^ permalink raw reply
* Re: [PATCH v17 01/47] llist: move llist_{head,node} definition to types.h
From: Byungchul Park @ 2025-10-16 0:46 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Mathieu Desnoyers, Greg Kroah-Hartman, linux-kernel, kernel_team,
Linus Torvalds, Damien Le Moal, linux-ide, Andreas Dilger,
linux-ext4, Ingo Molnar, Peter Zijlstra, Will Deacon,
Thomas Gleixner, Steven Rostedt, Joel Fernandes, Sasha Levin,
Daniel Vetter, duyuyang, Johannes Berg, Tejun Heo,
Theodore Ts'o, Matthew Wilcox, Dave Chinner, Amir Goldstein,
kernel-team, linux-mm, Andrew Morton, Michal Hocko, Minchan Kim,
Johannes Weiner, vdavydov.dev, SeongJae Park, jglisse,
Dennis Zhou, Christoph Lameter, Pekka Enberg, David Rientjes,
Vlastimil Babka, ngupta, linux-block, Josef Bacik, linux-fsdevel,
Jan Kara, Jeff Layton, Dan Williams, Christoph Hellwig,
Darrick J. Wong, dri-devel, rodrigosiqueiramelo, melissa.srw,
hamohammed.sa, harry.yoo, chris.p.wilson, Gwan-gyeong Mun,
max.byungchul.park, Boqun Feng, Waiman Long, yunseong.kim, ysk,
Yeoreum Yun, Netdev, Matthew Brost, her0gyugyu, Jonathan Corbet,
Catalin Marinas, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Andy Lutomirski, Sumit Semwal, gustavo,
Christian König, Andi Shyti, Lorenzo Stoakes,
Liam R. Howlett, Mike Rapoport, Suren Baghdasaryan,
Luis Chamberlain, Petr Pavlu, da.gomez, Sami Tolvanen,
Paul E. McKenney, Frederic Weisbecker, neeraj.upadhyay,
joelagnelf, Josh Triplett, Uladzislau Rezki (Sony), Lai Jiangshan,
qiang.zhang, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Benjamin Segall, Mel Gorman, Valentin Schneider, Chuck Lever,
neil, okorniev, Dai Ngo, Tom Talpey, trondmy, Anna Schumaker,
Kees Cook, Sebastian Andrzej Siewior, Clark Williams,
Mark Rutland, ada.coupriediaz, kristina.martsenko, Kefeng Wang,
Mark Brown, Kevin Brodsky, David Woodhouse, Shakeel Butt,
Alexei Starovoitov, Zi Yan, Yu Zhao, Baolin Wang, usamaarif642,
joel.granados, Wei Yang, Geert Uytterhoeven, tim.c.chen, linux,
Alexander Shishkin, lillian, Huacai Chen, francesco,
guoweikang.kernel, link, Josh Poimboeuf, Masahiro Yamada,
Christian Brauner, Thomas Weißschuh, Oleg Nesterov,
Mateusz Guzik, Andrii Nakryiko, wangfushuai, linux-doc,
linux-arm-kernel, linux-media, linaro-mm-sig, linux-i2c,
Linux-Arch, linux-modules, rcu, linux-nfs, linux-rt-devel
In-Reply-To: <3bbe14af-ccdc-4c78-a7ca-d4ed39fa6b5d@app.fastmail.com>
On Fri, Oct 03, 2025 at 01:19:33AM +0200, Arnd Bergmann wrote:
> On Thu, Oct 2, 2025, at 15:53, Mathieu Desnoyers wrote:
> > On 2025-10-02 04:24, Greg KH wrote:
> >> On Thu, Oct 02, 2025 at 05:12:01PM +0900, Byungchul Park wrote:
> >>> llist_head and llist_node can be used by some other header files. For
> >>> example, dept for tracking dependencies uses llist in its header. To
> >>> avoid header dependency, move them to types.h.
> >>
> >> If you need llist in your code, then include llist.h. Don't force all
> >> types.h users to do so as there is not a dependency in types.h for
> >> llist.h.
> >>
> >> This patch shouldn't be needed as you are hiding "header dependency" for
> >> other files.
> >
> > I agree that moving this into a catch-all types.h is not what we should
> > aim for.
> >
> > However, it's a good practice to move the type declarations to a
> > separate header file, so code that only cares about type and not
> > implementation of static inline functions can include just that.
> >
> > Perhaps we can move struct llist_head and struct llist_node to a new
> > include/linux/llist_types.h instead ?
>
> We have around a dozen types of linked lists, and the most common
> two of them are currently defined in linux/types.h, while the
> rest of them are each defined in the same header as the inteface
> definition.
>
> Duplicating each of those headers by splitting out the trivial
> type definition doesn't quite seem right either, as we'd end
> up with even more headers that have to be included indirectly
> in each compilation unit.
>
> Maybe a shared linux/list_types.h would work, to specifically
I found a way to resolve my issue, but I thought it's good idea
regardless of my issue and took a quick look. However, it seems like
there's an overwhelming amount of work since it might require to replace
all the existing include <linux/types.h> for use of list things with the
new one :-).
Byungchul
> contain all the list_head variants that are meant to be included
> in larger structures?
>
> Arnd
^ permalink raw reply
* Re: [PATCH v17 01/47] llist: move llist_{head,node} definition to types.h
From: Arnd Bergmann @ 2025-10-16 7:59 UTC (permalink / raw)
To: Byungchul Park
Cc: Mathieu Desnoyers, Greg Kroah-Hartman, linux-kernel, kernel_team,
Linus Torvalds, Damien Le Moal, linux-ide, Andreas Dilger,
linux-ext4, Ingo Molnar, Peter Zijlstra, Will Deacon,
Thomas Gleixner, Steven Rostedt, Joel Fernandes, Sasha Levin,
Daniel Vetter, duyuyang, Johannes Berg, Tejun Heo,
Theodore Ts'o, Matthew Wilcox, Dave Chinner, Amir Goldstein,
kernel-team, linux-mm, Andrew Morton, Michal Hocko, Minchan Kim,
Johannes Weiner, vdavydov.dev, SeongJae Park, jglisse,
Dennis Zhou, Christoph Lameter, Pekka Enberg, David Rientjes,
Vlastimil Babka, ngupta, linux-block, Josef Bacik, linux-fsdevel,
Jan Kara, Jeff Layton, Dan Williams, Christoph Hellwig,
Darrick J. Wong, dri-devel, rodrigosiqueiramelo, melissa.srw,
hamohammed.sa, harry.yoo, chris.p.wilson, Gwan-gyeong Mun,
max.byungchul.park, Boqun Feng, Waiman Long, yunseong.kim, ysk,
Yeoreum Yun, Netdev, Matthew Brost, her0gyugyu, Jonathan Corbet,
Catalin Marinas, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Andy Lutomirski, Sumit Semwal, gustavo,
Christian König, Andi Shyti, Lorenzo Stoakes,
Liam R. Howlett, Mike Rapoport, Suren Baghdasaryan,
Luis Chamberlain, Petr Pavlu, da.gomez, Sami Tolvanen,
Paul E. McKenney, Frederic Weisbecker, neeraj.upadhyay,
joelagnelf, Josh Triplett, Uladzislau Rezki (Sony), Lai Jiangshan,
qiang.zhang, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
Benjamin Segall, Mel Gorman, Valentin Schneider, Chuck Lever,
neil, okorniev, Dai Ngo, Tom Talpey, trondmy, Anna Schumaker,
Kees Cook, Sebastian Andrzej Siewior, Clark Williams,
Mark Rutland, ada.coupriediaz, kristina.martsenko, Kefeng Wang,
Mark Brown, Kevin Brodsky, David Woodhouse, Shakeel Butt,
Alexei Starovoitov, Zi Yan, Yu Zhao, Baolin Wang, usamaarif642,
joel.granados, Wei Yang, Geert Uytterhoeven, tim.c.chen, linux,
Alexander Shishkin, lillian, Huacai Chen, francesco,
guoweikang.kernel, link, Josh Poimboeuf, Masahiro Yamada,
Christian Brauner, Thomas Weißschuh, Oleg Nesterov,
Mateusz Guzik, Andrii Nakryiko, wangfushuai, linux-doc,
linux-arm-kernel, linux-media, linaro-mm-sig, linux-i2c,
Linux-Arch, linux-modules, rcu, linux-nfs, linux-rt-devel
In-Reply-To: <20251016004640.GB2948@system.software.com>
On Thu, Oct 16, 2025, at 02:46, Byungchul Park wrote:
> On Fri, Oct 03, 2025 at 01:19:33AM +0200, Arnd Bergmann wrote:
>> On Thu, Oct 2, 2025, at 15:53, Mathieu Desnoyers wrote:
>> > On 2025-10-02 04:24, Greg KH wrote:
>> >> On Thu, Oct 02, 2025 at 05:12:01PM +0900, Byungchul Park wrote:
>> Maybe a shared linux/list_types.h would work, to specifically
>
> I found a way to resolve my issue, but I thought it's good idea
> regardless of my issue and took a quick look. However, it seems like
> there's an overwhelming amount of work since it might require to replace
> all the existing include <linux/types.h> for use of list things with the
> new one :-).
I don't think it's that bad, since almost every header ends up
including linux/list.h indirectly at the moment.
A little bit of scripting to find the headers that reference
'struct list_head' but don't also include linux/list.h reveals
this relatively short set that would need to include the new
header:
> include/keys/asymmetric-parser.h
> include/linux/dynamic_debug.h
> include/linux/genalloc.h
> include/linux/gpio/machine.h
> include/linux/hiddev.h
> include/linux/iio/iio-opaque.h
> include/linux/iio/sysfs.h
> include/linux/input/touch-overlay.h
> include/linux/irq_poll.h
> include/linux/iscsi_boot_sysfs.h
> include/linux/kcore.h
> include/linux/kcsan-checks.h
> include/linux/kcsan.h
> include/linux/lockdep_types.h
> include/linux/logic_pio.h
> include/linux/maple.h
> include/linux/mfd/iqs62x.h
> include/linux/mlx5/macsec.h
> include/linux/mount.h
> include/linux/mtd/map.h
> include/linux/mtd/nand-qpic-common.h
> include/linux/mtd/partitions.h
> include/linux/mutex_types.h
> include/linux/nfs_fs_i.h
> include/linux/of_iommu.h
> include/linux/parport_pc.h
> include/linux/pinctrl/pinctrl.h
> include/linux/plist_types.h
> include/linux/pm_wakeup.h
> include/linux/reboot-mode.h
> include/linux/shm.h
> include/linux/smpboot.h
> include/linux/sunrpc/xprtmultipath.h
> include/linux/usb/audio.h
> include/linux/workqueue_types.h
> include/linux/zpool.h
> include/net/bluetooth/hci_sync.h
> include/net/bluetooth/l2cap.h
> include/net/bluetooth/rfcomm.h
> include/net/dcbnl.h
> include/sound/i2c.h
> include/sound/soc-jack.h
> include/target/iscsi/iscsi_transport.h
> include/video/udlfb.h
A lot of these don't have any #include statements at all,
which indicates that they expect to only be included in
places where the dependencies are already visible.
Arnd
^ permalink raw reply
* Re: [PATCH] docs: ABI: sysfs-module: list all taint flags
From: Petr Pavlu @ 2025-10-17 11:38 UTC (permalink / raw)
To: Randy Dunlap
Cc: Jonathan Corbet, linux-doc, Greg Kroah-Hartman, Luis Chamberlain,
Daniel Gomez, Sami Tolvanen, linux-kernel, linux-modules
In-Reply-To: <20251015221348.1125295-1-rdunlap@infradead.org>
On 10/16/25 12:13 AM, Randy Dunlap wrote:
> The list of module taint flags has not been updated lately as the
> taint flags list grows. Instead of trying to keep multiple lists
> updated, just refer to the list of kernel taint flags since they are
> the same.
>
> Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
> ---
> Cc: Jonathan Corbet <corbet@lwn.net>
> Cc: linux-doc@vger.kernel.org
> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Cc: Luis Chamberlain <mcgrof@kernel.org>
> Cc: Petr Pavlu <petr.pavlu@suse.com>
> Cc: Daniel Gomez <da.gomez@kernel.org>
> Cc: Sami Tolvanen <samitolvanen@google.com>
> Cc: linux-modules@vger.kernel.org
> ---
> Documentation/ABI/testing/sysfs-module | 10 ++--------
> Documentation/admin-guide/tainted-kernels.rst | 2 ++
> 2 files changed, 4 insertions(+), 8 deletions(-)
>
> --- linux-next-20251014.orig/Documentation/ABI/testing/sysfs-module
> +++ linux-next-20251014/Documentation/ABI/testing/sysfs-module
> @@ -52,14 +52,8 @@ What: /sys/module/*/taint
> Date: Jan 2012
> KernelVersion: 3.3
> Contact: Kay Sievers <kay.sievers@vrfy.org>
> -Description: Module taint flags:
> - == =====================
> - P proprietary module
> - O out-of-tree module
> - F force-loaded module
> - C staging driver module
> - E unsigned module
> - == =====================
> +Description: Module taint flags: same as the kernel taint flags.
> + See: :ref:`taint_flags` in Documentation/admin-guide/tainted-kernels.rst
The module taint flags that can appear in /sys/module/*/taint are
a subset of the kernel taint flags. By looking at the calls to
add_taint_module(), they are as follows:
Present:
TAINT_PROPRIETARY_MODULE
TAINT_OOT_MODULE
TAINT_FORCED_MODULE
TAINT_CRAP
TAINT_UNSIGNED_MODULE
Missing:
TAINT_LIVEPATCH
TAINT_TEST
+ potentially TEST_AUX.
Since this text specifically documents what can appear in
/sys/module/*/taint, I think we should still maintain a list of these
flags for accuracy.
Additionally, Documentation/admin-guide/tainted-kernels.rst provides
taint descriptions for the kernel as a whole, which can be misleading
for individual modules. For instance, for TAINT_LIVEPATCH, the document
says "kernel has been live patched", but in the context of
/sys/module/*/taint, it means "this is a livepatch module".
--
Thanks,
Petr
^ permalink raw reply
* Re: [PATCH 1/2] module: Override -EEXISTS module return
From: Petr Pavlu @ 2025-10-17 12:04 UTC (permalink / raw)
To: Lucas De Marchi; +Cc: linux-modules, linux-kernel
In-Reply-To: <20251013-module-warn-ret-v1-1-ab65b41af01f@intel.com>
On 10/13/25 6:26 PM, Lucas De Marchi wrote:
> The -EEXIST errno is reserved by the module loading functionality. When
> userspace calls [f]init_module(), it expects a -EEXIST to mean that the
> module is already loaded in the kernel. If module_init() returns it,
> that is not true anymore.
>
> Add a warning and override the return code to workaround modules
> currently returning the wrong code. It's expected that they eventually
> migrate to a better suited error.
>
> Closes: https://lore.kernel.org/all/aKLzsAX14ybEjHfJ@orbyte.nwl.cc/
> Signed-off-by: Lucas De Marchi <lucas.demarchi@intel.com>
> ---
> kernel/module/main.c | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/kernel/module/main.c b/kernel/module/main.c
> index c66b261849362..74ff87b13c517 100644
> --- a/kernel/module/main.c
> +++ b/kernel/module/main.c
> @@ -3038,6 +3038,11 @@ static noinline int do_init_module(struct module *mod)
> if (mod->init != NULL)
> ret = do_one_initcall(mod->init);
> if (ret < 0) {
> + if (ret == -EEXIST) {
> + pr_warn("%s: init suspiciously returned -EEXIST: Overriding with -EBUSY\n",
> + mod->name);
> + ret = -EBUSY;
> + }
> goto fail_free_freeinit;
> }
> if (ret > 0) {
>
I assume you intentionally omitted the "(reserved for loaded modules)"
part from my previous suggestion [1] to keep the error message short.
Looks ok to me then.
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
[1] https://lore.kernel.org/linux-modules/ce7f293c-d9f9-4137-bcad-8cc492d34773@suse.com/
--
Thanks,
Petr
^ permalink raw reply
* Re: [PATCH 2/2] module: Simplify warning on positive returns from module_init()
From: Petr Pavlu @ 2025-10-17 12:06 UTC (permalink / raw)
To: Lucas De Marchi; +Cc: linux-modules, linux-kernel
In-Reply-To: <20251013-module-warn-ret-v1-2-ab65b41af01f@intel.com>
On 10/13/25 6:26 PM, Lucas De Marchi wrote:
> It should now be rare to trigger this warning - it doesn't need to be so
> verbose. Make it follow the usual style in the module loading code.
>
> For the same reason, drop the dump_stack().
>
> Suggested-by: Petr Pavlu <petr.pavlu@suse.com>
> Signed-off-by: Lucas De Marchi <lucas.demarchi@intel.com>
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
--
Thanks,
Petr
^ permalink raw reply
* [PATCH v6 00/17] lib/crypto: Move SHA3 to lib/crypto, add SHAKE* and add ML-DSA signing
From: David Howells @ 2025-10-17 14:42 UTC (permalink / raw)
To: Eric Biggers
Cc: David Howells, Jason A . Donenfeld, Ard Biesheuvel, Herbert Xu,
Stephan Mueller, Lukas Wunner, Ignat Korchagin, Luis Chamberlain,
Petr Pavlu, Daniel Gomez, Sami Tolvanen, linux-crypto, keyrings,
linux-modules, linux-kernel
Hi Eric, Herbert, et al.,
Here's a set of patches does the following:
Firstly, SHA-3 is implemented in lib/crypto and SHAKE* support is added:
(1) Renames s390 and arm64 sha3_* functions to avoid name collisions.
(2) Copies the core of SHA3 support from crypto/ to lib/crypto/.
(3) Simplifies the internal code to maintain the buffer in little endian
form, thereby simplifying the update and extraction code which don't
then need to worry about this. Instead, the state buffer is
byteswapped before and after.
(4) Moves the Iota transform into the function with the rest of the
transforms.
(5) Adds SHAKE128 and SHAKE256 support (needed for ML-DSA).
(6) Adds a kunit test for SHA3 in lib/crypto/tests/.
(7) Adds API documentation for SHA3.
Note that only the generic code is moved across; the asm-optimised stuff is
not touched as I'm not familiar with that.
Secondly, SHA-3 crypto/ support is added, though this turns out not to be
necessary for ML-DSA:
(8) Make the jitterentropy random number generator use lib/crypto sha-3.
I can't get it to work with crypto/sha3_generic when that is modified
to use lib/crypto. It looks like there's some race that prevents the
kernel from booting, but I can't tell what and it's really hard to
debug. However, it works fine with lib/crypto/sha3 directly.
(9) Make crypto/sha3_generic use lib/crypto/sha3.
(10) Add SHAKE-128 and SHAKE-256 crypto_sig support, generating 16-byte and
32-byte fixed output respectively. The XOF features aren't available
through this.
Thirdly, add ML-DSA support and allow module signing to use it. This needs
some more code cleanup, but it does work. OpenSSL doesn't support
CMS_NOATTR with ML-DSA, unfortunately, so that has to be turned off for
ML-DSA.
(11) Add ML-DSA signing and signature verification code, extracted from
Stephan Mueller's Leancrypto project. This needs more code cleanup,
but I think it's worth posting it now. I also suspect I should remove
the in-kernel signature generation side of things and only keep the
verification side. It is accessed through crypto_sig.
(12) Add ML-DSA keypair generation code. I don't think we want this.
(13) Add a kunit test in three installments (due to size) to add some
testing for the three different levels of ML-DSA (44, 65 and 87).
(14) Modify PKCS#7 support to allow kernel module signatures to carry
authenticatedAttributes as OpenSSL refuses to let them be opted out of
for ML-DSA (CMS_NOATTR). This adds an extra digest calculation to the
process.
(15) Modify PKCS#7 to pass the authenticatedAttributes directly to the
ML-DSA algorithm rather than passing over a digest as is done with RSA
as ML-DSA wants to do its own hashing and will add other stuff into
the hash. We could use hashML-DSA or an external mu instead, but they
aren't standardised for CMS yet.
(16) Add support to the PKCS#7 and X.509 parsers for ML-DSA.
(17) Modify sign-file to handle OpenSSL not permitting CMS_NOATTR with
ML-DSA.
(18) Allow SHA-3 algorithms, including SHAKE256, to be used for the message
digest and add ML-DSA to the choice of algorithm with which to sign.
With that, ML-DSA signing appears to work.
This is based on Eric's libcrypto-next branch.
The patches can also be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=keys-pqc
David
Changes
=======
ver #6)
- Added a patch to make the jitterentropy RNG use lib/sha3.
- Added back the crypto/sha3_generic changes.
- Added ML-DSA implementation (still needs more cleanup).
- Added kunit test for ML-DSA.
- Modified PKCS#7 to accommodate ML-DSA.
- Modified PKCS#7 and X.509 to allow ML-DSA to be specified and used.
- Modified sign-file to not use CMS_NOATTR with ML-DSA.
- Allowed SHA3 and SHAKE* algorithms for module signing default.
- Allowed ML-DSA-{44,65,87} to be selected as the module signing default.
ver #5)
- Fix gen-hash-testvecs.py to correctly handle algo names that contain a
dash.
- Fix gen-hash-testvecs.py to not generate HMAC for SHA3-* or SHAKE* as
these don't currently have HMAC variants implemented.
- Fix algo names to be correct.
- Fix kunit module description as it now tests all SHA3 variants.
ver #4)
- Fix a couple of arm64 build problems.
- Doc fixes:
- Fix the description of the algorithm to be closer to the NIST spec's
terminology.
- Don't talk of finialising the context for XOFs.
- Don't say "Return: None".
- Declare the "Context" to be "Any context" and make no mention of the
fact that it might use the FPU.
- Change "initialise" to "initialize".
- Don't warn that the context is relatively large for stack use.
- Use size_t for size parameters/variables.
- Make the module_exit unconditional.
- Dropped the crypto/ dir-affecting patches for the moment.
ver #3)
- Renamed conflicting arm64 functions.
- Made a separate wrapper API for each algorithm in the family.
- Removed sha3_init(), sha3_reinit() and sha3_final().
- Removed sha3_ctx::digest_size.
- Renamed sha3_ctx::partial to sha3_ctx::absorb_offset.
- Refer to the output of SHAKE* as "output" not "digest".
- Moved the Iota transform into the one-round function.
- Made sha3_update() warn if called after sha3_squeeze().
- Simplified the module-load test to not do update after squeeze.
- Added Return: and Context: kdoc statements and expanded the kdoc
headers.
- Added an API description document.
- Overhauled the kunit tests.
- Only have one kunit test.
- Only call the general hash tester on one algo.
- Add separate simple cursory checks for the other algos.
- Add resqueezing tests.
- Add some NIST example tests.
- Changed crypto/sha3_generic to use this
- Added SHAKE128/256 to crypto/sha3_generic and crypto/testmgr
- Folded struct sha3_state into struct sha3_ctx.
ver #2)
- Simplify the endianness handling.
- Rename sha3_final() to sha3_squeeze() and don't clear the context at the
end as it's permitted to continue calling sha3_final() to extract
continuations of the digest (needed by ML-DSA).
- Don't reapply the end marker to the hash state in continuation
sha3_squeeze() unless sha3_update() gets called again (needed by
ML-DSA).
- Give sha3_squeeze() the amount of digest to produce as a parameter
rather than using ctx->digest_size and don't return the amount digested.
- Reimplement sha3_final() as a wrapper around sha3_squeeze() that
extracts ctx->digest_size amount of digest and then zeroes out the
context. The latter is necessary to avoid upsetting
hash-test-template.h.
- Provide a sha3_reinit() function to clear the state, but to leave the
parameters that indicate the hash properties unaffected, allowing for
reuse.
- Provide a sha3_set_digestsize() function to change the size of the
digest to be extracted by sha3_final(). sha3_squeeze() takes a
parameter for this instead.
- Don't pass the digest size as a parameter to shake128/256_init() but
rather default to 128/256 bits as per the function name.
- Provide a sha3_clear() function to zero out the context.
David Howells (17):
s390/sha3: Rename conflicting functions
arm64/sha3: Rename conflicting functions
lib/crypto: Add SHA3-224, SHA3-256, SHA3-384, SHA3-512, SHAKE128,
SHAKE256
lib/crypto: Move the SHA3 Iota transform into the single round
function
lib/crypto: Add SHA3 kunit tests
crypto/jitterentropy: Use lib/crypto/sha3
crypto/sha3: Use lib/crypto/sha3
crypto/sha3: Add SHAKE128/256 support
crypto: SHAKE tests
crypto: Add ML-DSA/Dilithium support
crypto: Add ML-DSA/Dilithium keypair generation support
crypto: Add ML-DSA-44 pure rejection test vectors as a kunit test
crypto: Add ML-DSA-65 pure rejection test vectors as a kunit test
crypto: Add ML-DSA-87 pure rejection test vectors as a kunit test
pkcs7: Allow the signing algo to calculate the digest itself
pkcs7, x509: Add ML-DSA support
modsign: Enable ML-DSA module signing
Documentation/admin-guide/module-signing.rst | 15 +-
Documentation/crypto/index.rst | 1 +
Documentation/crypto/sha3.rst | 237 +
arch/arm64/crypto/sha3-ce-glue.c | 47 +-
arch/s390/crypto/sha3_256_s390.c | 26 +-
arch/s390/crypto/sha3_512_s390.c | 26 +-
certs/Kconfig | 24 +
certs/Makefile | 3 +
crypto/Kconfig | 1 +
crypto/Makefile | 1 +
crypto/asymmetric_keys/pkcs7_parser.c | 19 +-
crypto/asymmetric_keys/pkcs7_verify.c | 52 +-
crypto/asymmetric_keys/public_key.c | 7 +
crypto/asymmetric_keys/x509_cert_parser.c | 24 +
crypto/jitterentropy-kcapi.c | 100 +-
crypto/jitterentropy.c | 7 +-
crypto/jitterentropy.h | 8 +-
crypto/ml_dsa/Kconfig | 32 +
crypto/ml_dsa/Makefile | 20 +
crypto/ml_dsa/dilithium.h | 804 ++
crypto/ml_dsa/dilithium_44.c | 33 +
crypto/ml_dsa/dilithium_44.h | 412 +
crypto/ml_dsa/dilithium_65.c | 33 +
crypto/ml_dsa/dilithium_65.h | 412 +
crypto/ml_dsa/dilithium_87.c | 33 +
crypto/ml_dsa/dilithium_87.h | 412 +
crypto/ml_dsa/dilithium_api.c | 821 ++
crypto/ml_dsa/dilithium_debug.h | 80 +
crypto/ml_dsa/dilithium_ntt.c | 89 +
crypto/ml_dsa/dilithium_ntt.h | 35 +
crypto/ml_dsa/dilithium_pack.h | 285 +
crypto/ml_dsa/dilithium_pct.h | 53 +
crypto/ml_dsa/dilithium_poly.c | 586 +
crypto/ml_dsa/dilithium_poly.h | 190 +
crypto/ml_dsa/dilithium_poly_c.h | 149 +
crypto/ml_dsa/dilithium_poly_common.h | 35 +
crypto/ml_dsa/dilithium_polyvec.h | 363 +
crypto/ml_dsa/dilithium_polyvec_c.h | 114 +
.../dilithium_pure_rejection_vectors_44.h | 812 ++
.../dilithium_pure_rejection_vectors_65.h | 8290 ++++++++++++
.../dilithium_pure_rejection_vectors_87.h | 10761 ++++++++++++++++
crypto/ml_dsa/dilithium_reduce.h | 108 +
crypto/ml_dsa/dilithium_rounding.c | 128 +
crypto/ml_dsa/dilithium_rounding.h | 45 +
crypto/ml_dsa/dilithium_selftest.c | 183 +
crypto/ml_dsa/dilithium_service_helpers.h | 99 +
crypto/ml_dsa/dilithium_sig.c | 404 +
crypto/ml_dsa/dilithium_signature_c.c | 198 +
crypto/ml_dsa/dilithium_signature_c.h | 60 +
crypto/ml_dsa/dilithium_signature_helper.c | 110 +
crypto/ml_dsa/dilithium_signature_impl.h | 1033 ++
crypto/ml_dsa/dilithium_type.h | 281 +
crypto/ml_dsa/dilithium_zetas.c | 67 +
crypto/ml_dsa/fips_mode.h | 45 +
crypto/ml_dsa/signature_domain_separation.c | 213 +
crypto/ml_dsa/signature_domain_separation.h | 33 +
crypto/ml_dsa/small_stack_support.h | 40 +
crypto/sha3_generic.c | 241 +-
crypto/testmgr.c | 14 +
crypto/testmgr.h | 59 +
include/crypto/algapi.h | 2 +-
include/crypto/public_key.h | 1 +
include/crypto/sha3.h | 433 +-
include/linux/oid_registry.h | 5 +
kernel/module/Kconfig | 5 +
lib/crypto/Kconfig | 7 +
lib/crypto/Makefile | 6 +
lib/crypto/sha3.c | 511 +
lib/crypto/tests/Kconfig | 11 +
lib/crypto/tests/Makefile | 1 +
lib/crypto/tests/sha3_kunit.c | 342 +
lib/crypto/tests/sha3_testvecs.h | 231 +
scripts/crypto/gen-hash-testvecs.py | 10 +-
scripts/sign-file.c | 26 +-
74 files changed, 30050 insertions(+), 354 deletions(-)
create mode 100644 Documentation/crypto/sha3.rst
create mode 100644 crypto/ml_dsa/Kconfig
create mode 100644 crypto/ml_dsa/Makefile
create mode 100644 crypto/ml_dsa/dilithium.h
create mode 100644 crypto/ml_dsa/dilithium_44.c
create mode 100644 crypto/ml_dsa/dilithium_44.h
create mode 100644 crypto/ml_dsa/dilithium_65.c
create mode 100644 crypto/ml_dsa/dilithium_65.h
create mode 100644 crypto/ml_dsa/dilithium_87.c
create mode 100644 crypto/ml_dsa/dilithium_87.h
create mode 100644 crypto/ml_dsa/dilithium_api.c
create mode 100644 crypto/ml_dsa/dilithium_debug.h
create mode 100644 crypto/ml_dsa/dilithium_ntt.c
create mode 100644 crypto/ml_dsa/dilithium_ntt.h
create mode 100644 crypto/ml_dsa/dilithium_pack.h
create mode 100644 crypto/ml_dsa/dilithium_pct.h
create mode 100644 crypto/ml_dsa/dilithium_poly.c
create mode 100644 crypto/ml_dsa/dilithium_poly.h
create mode 100644 crypto/ml_dsa/dilithium_poly_c.h
create mode 100644 crypto/ml_dsa/dilithium_poly_common.h
create mode 100644 crypto/ml_dsa/dilithium_polyvec.h
create mode 100644 crypto/ml_dsa/dilithium_polyvec_c.h
create mode 100644 crypto/ml_dsa/dilithium_pure_rejection_vectors_44.h
create mode 100644 crypto/ml_dsa/dilithium_pure_rejection_vectors_65.h
create mode 100644 crypto/ml_dsa/dilithium_pure_rejection_vectors_87.h
create mode 100644 crypto/ml_dsa/dilithium_reduce.h
create mode 100644 crypto/ml_dsa/dilithium_rounding.c
create mode 100644 crypto/ml_dsa/dilithium_rounding.h
create mode 100644 crypto/ml_dsa/dilithium_selftest.c
create mode 100644 crypto/ml_dsa/dilithium_service_helpers.h
create mode 100644 crypto/ml_dsa/dilithium_sig.c
create mode 100644 crypto/ml_dsa/dilithium_signature_c.c
create mode 100644 crypto/ml_dsa/dilithium_signature_c.h
create mode 100644 crypto/ml_dsa/dilithium_signature_helper.c
create mode 100644 crypto/ml_dsa/dilithium_signature_impl.h
create mode 100644 crypto/ml_dsa/dilithium_type.h
create mode 100644 crypto/ml_dsa/dilithium_zetas.c
create mode 100644 crypto/ml_dsa/fips_mode.h
create mode 100644 crypto/ml_dsa/signature_domain_separation.c
create mode 100644 crypto/ml_dsa/signature_domain_separation.h
create mode 100644 crypto/ml_dsa/small_stack_support.h
create mode 100644 lib/crypto/sha3.c
create mode 100644 lib/crypto/tests/sha3_kunit.c
create mode 100644 lib/crypto/tests/sha3_testvecs.h
^ permalink raw reply
* [PATCH v6 01/17] s390/sha3: Rename conflicting functions
From: David Howells @ 2025-10-17 14:42 UTC (permalink / raw)
To: Eric Biggers
Cc: David Howells, Jason A . Donenfeld, Ard Biesheuvel, Herbert Xu,
Stephan Mueller, Lukas Wunner, Ignat Korchagin, Luis Chamberlain,
Petr Pavlu, Daniel Gomez, Sami Tolvanen, linux-crypto, keyrings,
linux-modules, linux-kernel, Harald Freudenberger, Holger Dengler,
linux-s390
In-Reply-To: <20251017144311.817771-1-dhowells@redhat.com>
Rename the s390 sha3_* functions to have an "s390_" prefix to avoid
conflict with generic code.
Signed-off-by: David Howells <dhowells@redhat.com>
Acked-By: Harald Freudenberger <freude@linux.ibm.com>
cc: Eric Biggers <ebiggers@kernel.org>
cc: Jason A. Donenfeld <Jason@zx2c4.com>
cc: Ard Biesheuvel <ardb@kernel.org>
cc: Holger Dengler <dengler@linux.ibm.com>
cc: Herbert Xu <herbert@gondor.apana.org.au>
cc: Stephan Mueller <smueller@chronox.de>
cc: linux-crypto@vger.kernel.org
cc: linux-s390@vger.kernel.org
---
arch/s390/crypto/sha3_256_s390.c | 26 +++++++++++++-------------
arch/s390/crypto/sha3_512_s390.c | 26 +++++++++++++-------------
2 files changed, 26 insertions(+), 26 deletions(-)
diff --git a/arch/s390/crypto/sha3_256_s390.c b/arch/s390/crypto/sha3_256_s390.c
index 03bb4f4bab70..fd5ecae60a57 100644
--- a/arch/s390/crypto/sha3_256_s390.c
+++ b/arch/s390/crypto/sha3_256_s390.c
@@ -19,7 +19,7 @@
#include "sha.h"
-static int sha3_256_init(struct shash_desc *desc)
+static int s390_sha3_256_init(struct shash_desc *desc)
{
struct s390_sha_ctx *sctx = shash_desc_ctx(desc);
@@ -32,7 +32,7 @@ static int sha3_256_init(struct shash_desc *desc)
return 0;
}
-static int sha3_256_export(struct shash_desc *desc, void *out)
+static int s390_sha3_256_export(struct shash_desc *desc, void *out)
{
struct s390_sha_ctx *sctx = shash_desc_ctx(desc);
union {
@@ -50,7 +50,7 @@ static int sha3_256_export(struct shash_desc *desc, void *out)
return 0;
}
-static int sha3_256_import(struct shash_desc *desc, const void *in)
+static int s390_sha3_256_import(struct shash_desc *desc, const void *in)
{
struct s390_sha_ctx *sctx = shash_desc_ctx(desc);
union {
@@ -68,22 +68,22 @@ static int sha3_256_import(struct shash_desc *desc, const void *in)
return 0;
}
-static int sha3_224_import(struct shash_desc *desc, const void *in)
+static int s390_sha3_224_import(struct shash_desc *desc, const void *in)
{
struct s390_sha_ctx *sctx = shash_desc_ctx(desc);
- sha3_256_import(desc, in);
+ s390_sha3_256_import(desc, in);
sctx->func = CPACF_KIMD_SHA3_224;
return 0;
}
static struct shash_alg sha3_256_alg = {
.digestsize = SHA3_256_DIGEST_SIZE, /* = 32 */
- .init = sha3_256_init,
+ .init = s390_sha3_256_init,
.update = s390_sha_update_blocks,
.finup = s390_sha_finup,
- .export = sha3_256_export,
- .import = sha3_256_import,
+ .export = s390_sha3_256_export,
+ .import = s390_sha3_256_import,
.descsize = S390_SHA_CTX_SIZE,
.statesize = SHA3_STATE_SIZE,
.base = {
@@ -96,22 +96,22 @@ static struct shash_alg sha3_256_alg = {
}
};
-static int sha3_224_init(struct shash_desc *desc)
+static int s390_sha3_224_init(struct shash_desc *desc)
{
struct s390_sha_ctx *sctx = shash_desc_ctx(desc);
- sha3_256_init(desc);
+ s390_sha3_256_init(desc);
sctx->func = CPACF_KIMD_SHA3_224;
return 0;
}
static struct shash_alg sha3_224_alg = {
.digestsize = SHA3_224_DIGEST_SIZE,
- .init = sha3_224_init,
+ .init = s390_sha3_224_init,
.update = s390_sha_update_blocks,
.finup = s390_sha_finup,
- .export = sha3_256_export, /* same as for 256 */
- .import = sha3_224_import, /* function code different! */
+ .export = s390_sha3_256_export, /* same as for 256 */
+ .import = s390_sha3_224_import, /* function code different! */
.descsize = S390_SHA_CTX_SIZE,
.statesize = SHA3_STATE_SIZE,
.base = {
diff --git a/arch/s390/crypto/sha3_512_s390.c b/arch/s390/crypto/sha3_512_s390.c
index a5c9690eecb1..f4b52a3a0433 100644
--- a/arch/s390/crypto/sha3_512_s390.c
+++ b/arch/s390/crypto/sha3_512_s390.c
@@ -18,7 +18,7 @@
#include "sha.h"
-static int sha3_512_init(struct shash_desc *desc)
+static int s390_sha3_512_init(struct shash_desc *desc)
{
struct s390_sha_ctx *sctx = shash_desc_ctx(desc);
@@ -31,7 +31,7 @@ static int sha3_512_init(struct shash_desc *desc)
return 0;
}
-static int sha3_512_export(struct shash_desc *desc, void *out)
+static int s390_sha3_512_export(struct shash_desc *desc, void *out)
{
struct s390_sha_ctx *sctx = shash_desc_ctx(desc);
union {
@@ -49,7 +49,7 @@ static int sha3_512_export(struct shash_desc *desc, void *out)
return 0;
}
-static int sha3_512_import(struct shash_desc *desc, const void *in)
+static int s390_sha3_512_import(struct shash_desc *desc, const void *in)
{
struct s390_sha_ctx *sctx = shash_desc_ctx(desc);
union {
@@ -67,22 +67,22 @@ static int sha3_512_import(struct shash_desc *desc, const void *in)
return 0;
}
-static int sha3_384_import(struct shash_desc *desc, const void *in)
+static int s390_sha3_384_import(struct shash_desc *desc, const void *in)
{
struct s390_sha_ctx *sctx = shash_desc_ctx(desc);
- sha3_512_import(desc, in);
+ s390_sha3_512_import(desc, in);
sctx->func = CPACF_KIMD_SHA3_384;
return 0;
}
static struct shash_alg sha3_512_alg = {
.digestsize = SHA3_512_DIGEST_SIZE,
- .init = sha3_512_init,
+ .init = s390_sha3_512_init,
.update = s390_sha_update_blocks,
.finup = s390_sha_finup,
- .export = sha3_512_export,
- .import = sha3_512_import,
+ .export = s390_sha3_512_export,
+ .import = s390_sha3_512_import,
.descsize = S390_SHA_CTX_SIZE,
.statesize = SHA3_STATE_SIZE,
.base = {
@@ -97,22 +97,22 @@ static struct shash_alg sha3_512_alg = {
MODULE_ALIAS_CRYPTO("sha3-512");
-static int sha3_384_init(struct shash_desc *desc)
+static int s390_sha3_384_init(struct shash_desc *desc)
{
struct s390_sha_ctx *sctx = shash_desc_ctx(desc);
- sha3_512_init(desc);
+ s390_sha3_512_init(desc);
sctx->func = CPACF_KIMD_SHA3_384;
return 0;
}
static struct shash_alg sha3_384_alg = {
.digestsize = SHA3_384_DIGEST_SIZE,
- .init = sha3_384_init,
+ .init = s390_sha3_384_init,
.update = s390_sha_update_blocks,
.finup = s390_sha_finup,
- .export = sha3_512_export, /* same as for 512 */
- .import = sha3_384_import, /* function code different! */
+ .export = s390_sha3_512_export, /* same as for 512 */
+ .import = s390_sha3_384_import, /* function code different! */
.descsize = S390_SHA_CTX_SIZE,
.statesize = SHA3_STATE_SIZE,
.base = {
^ permalink raw reply related
* [PATCH v6 02/17] arm64/sha3: Rename conflicting functions
From: David Howells @ 2025-10-17 14:42 UTC (permalink / raw)
To: Eric Biggers
Cc: David Howells, Jason A . Donenfeld, Ard Biesheuvel, Herbert Xu,
Stephan Mueller, Lukas Wunner, Ignat Korchagin, Luis Chamberlain,
Petr Pavlu, Daniel Gomez, Sami Tolvanen, linux-crypto, keyrings,
linux-modules, linux-kernel, Catalin Marinas, Will Deacon,
linux-arm-kernel
In-Reply-To: <20251017144311.817771-1-dhowells@redhat.com>
Rename the arm64 sha3_* functions to have an "arm64_" prefix to avoid
conflict with generic code.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Eric Biggers <ebiggers@kernel.org>
cc: Jason A. Donenfeld <Jason@zx2c4.com>
cc: Ard Biesheuvel <ardb@kernel.org>
cc: Catalin Marinas <catalin.marinas@arm.com>
cc: Will Deacon <will@kernel.org>
cc: Herbert Xu <herbert@gondor.apana.org.au>
cc: Stephan Mueller <smueller@chronox.de>
cc: linux-crypto@vger.kernel.org
cc: linux-arm-kernel@lists.infradead.org
---
arch/arm64/crypto/sha3-ce-glue.c | 22 +++++++++++-----------
1 file changed, 11 insertions(+), 11 deletions(-)
diff --git a/arch/arm64/crypto/sha3-ce-glue.c b/arch/arm64/crypto/sha3-ce-glue.c
index b4f1001046c9..426d8044535a 100644
--- a/arch/arm64/crypto/sha3-ce-glue.c
+++ b/arch/arm64/crypto/sha3-ce-glue.c
@@ -31,7 +31,7 @@ MODULE_ALIAS_CRYPTO("sha3-512");
asmlinkage int sha3_ce_transform(u64 *st, const u8 *data, int blocks,
int md_len);
-static int sha3_update(struct shash_desc *desc, const u8 *data,
+static int arm64_sha3_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
struct sha3_state *sctx = shash_desc_ctx(desc);
@@ -55,8 +55,8 @@ static int sha3_update(struct shash_desc *desc, const u8 *data,
return len;
}
-static int sha3_finup(struct shash_desc *desc, const u8 *src, unsigned int len,
- u8 *out)
+static int arm64_sha3_finup(struct shash_desc *desc, const u8 *src, unsigned int len,
+ u8 *out)
{
struct sha3_state *sctx = shash_desc_ctx(desc);
struct crypto_shash *tfm = desc->tfm;
@@ -90,8 +90,8 @@ static int sha3_finup(struct shash_desc *desc, const u8 *src, unsigned int len,
static struct shash_alg algs[] = { {
.digestsize = SHA3_224_DIGEST_SIZE,
.init = crypto_sha3_init,
- .update = sha3_update,
- .finup = sha3_finup,
+ .update = arm64_sha3_update,
+ .finup = arm64_sha3_finup,
.descsize = SHA3_STATE_SIZE,
.base.cra_name = "sha3-224",
.base.cra_driver_name = "sha3-224-ce",
@@ -102,8 +102,8 @@ static struct shash_alg algs[] = { {
}, {
.digestsize = SHA3_256_DIGEST_SIZE,
.init = crypto_sha3_init,
- .update = sha3_update,
- .finup = sha3_finup,
+ .update = arm64_sha3_update,
+ .finup = arm64_sha3_finup,
.descsize = SHA3_STATE_SIZE,
.base.cra_name = "sha3-256",
.base.cra_driver_name = "sha3-256-ce",
@@ -114,8 +114,8 @@ static struct shash_alg algs[] = { {
}, {
.digestsize = SHA3_384_DIGEST_SIZE,
.init = crypto_sha3_init,
- .update = sha3_update,
- .finup = sha3_finup,
+ .update = arm64_sha3_update,
+ .finup = arm64_sha3_finup,
.descsize = SHA3_STATE_SIZE,
.base.cra_name = "sha3-384",
.base.cra_driver_name = "sha3-384-ce",
@@ -126,8 +126,8 @@ static struct shash_alg algs[] = { {
}, {
.digestsize = SHA3_512_DIGEST_SIZE,
.init = crypto_sha3_init,
- .update = sha3_update,
- .finup = sha3_finup,
+ .update = arm64_sha3_update,
+ .finup = arm64_sha3_finup,
.descsize = SHA3_STATE_SIZE,
.base.cra_name = "sha3-512",
.base.cra_driver_name = "sha3-512-ce",
^ permalink raw reply related
* [PATCH v6 03/17] lib/crypto: Add SHA3-224, SHA3-256, SHA3-384, SHA3-512, SHAKE128, SHAKE256
From: David Howells @ 2025-10-17 14:42 UTC (permalink / raw)
To: Eric Biggers
Cc: David Howells, Jason A . Donenfeld, Ard Biesheuvel, Herbert Xu,
Stephan Mueller, Lukas Wunner, Ignat Korchagin, Luis Chamberlain,
Petr Pavlu, Daniel Gomez, Sami Tolvanen, linux-crypto, keyrings,
linux-modules, linux-kernel
In-Reply-To: <20251017144311.817771-1-dhowells@redhat.com>
Add SHA3, providing SHA3-224, SHA3-256, SHA3-384, SHA3-512, SHAKE128 and
SHAKE256 to lib/crypto.
Notes:
(1) I've left hooks in sha3.c for asm-optimised variants, but as I don't
entirely know what those might look like, not having implemented any,
the hooks' usability is uncertain.
(2) The SHAKE algorithms will be required for ML-DSA.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Eric Biggers <ebiggers@kernel.org>
cc: Jason A. Donenfeld <Jason@zx2c4.com>
cc: Ard Biesheuvel <ardb@kernel.org>
cc: Herbert Xu <herbert@gondor.apana.org.au>
cc: Stephan Mueller <smueller@chronox.de>
cc: linux-crypto@vger.kernel.org
Changes
=======
v4)
- Doc fixes:
- Fix the description of the algorithm to be closer to the NIST spec's
terminology.
- Don't talk of finialising the context for XOFs.
- Don't say "Return: None".
- Declare the "Context" to be "Any context" and make no mention of the
fact that it might use the FPU.
- Change "initialise" to "initialize".
- Don't warn that the context is relatively large for stack use.
- Use size_t for size parameters/variables.
- Make the module_exit unconditional.
v3)
- Split the s390 function rename out to a separate patch.
v2)
- Simplify things by keeping the state array in LE form and byteswapping
all the 64-bit words before and after applying the keccak function on a
BE system. This means no byteswapping is required when XOR'ing data
into the state array or when extracting the digest. Further, this is a
no-op on LE systems.
- Rename sha3_final() to sha3_squeeze() and don't clear the context at the
end as it's permitted to continue calling sha3_final() to extract
continuations of the digest (needed by ML-DSA).
- Don't reapply the end marker to the hash state in continuation
sha3_squeeze() unless sha3_update() gets called again (needed by
ML-DSA).
- Give sha3_squeeze() the amount of digest to produce as a parameter
rather than using ctx->digest_size and don't return the amount digested.
- Reimplement sha3_final() as a wrapper around sha3_squeeze() that
extracts ctx->digest_size amount of digest and then zeroes out the
context. The latter is necessary to avoid upsetting
hash-test-template.h.
- Provide a sha3_reinit() function to clear the state, but to leave the
parameters that indicate the hash properties unaffected, allowing for
reuse.
- Provide a sha3_set_digestsize() function to change the size of the
digest to be extracted by sha3_final(). sha3_squeeze() takes a
parameter for this instead.
- Don't pass the digest size as a parameter to shake128/256_init() but
rather default to 128/256 bits as per the function name.
- Provide a sha3_clear() function to zero out the context.
---
Documentation/crypto/index.rst | 1 +
Documentation/crypto/sha3.rst | 241 ++++++++++++++++
include/crypto/sha3.h | 433 +++++++++++++++++++++++++++-
lib/crypto/Kconfig | 7 +
lib/crypto/Makefile | 6 +
lib/crypto/sha3.c | 511 +++++++++++++++++++++++++++++++++
6 files changed, 1198 insertions(+), 1 deletion(-)
create mode 100644 Documentation/crypto/sha3.rst
create mode 100644 lib/crypto/sha3.c
diff --git a/Documentation/crypto/index.rst b/Documentation/crypto/index.rst
index 100b47d049c0..4ee667c446f9 100644
--- a/Documentation/crypto/index.rst
+++ b/Documentation/crypto/index.rst
@@ -27,3 +27,4 @@ for cryptographic use cases, as well as programming examples.
descore-readme
device_drivers/index
krb5
+ sha3
diff --git a/Documentation/crypto/sha3.rst b/Documentation/crypto/sha3.rst
new file mode 100644
index 000000000000..3255618ea2ec
--- /dev/null
+++ b/Documentation/crypto/sha3.rst
@@ -0,0 +1,241 @@
+.. SPDX-License-Identifier: GPL-2.0-or-later
+
+==========================
+SHA-3 Algorithm collection
+==========================
+
+.. Contents:
+
+ - Overview
+ - Basic API
+ - Extendable-Output Functions
+ - Convenience API
+ - Internal API
+ - Testing
+ - References
+ - API Function Reference
+
+
+Overview
+========
+
+The SHA-3 algorithm base, as specified in NIST FIPS-202[1], provides a number
+of specific variants all based on the same basic algorithm (the Keccak sponge
+function and permutation). The differences between them are: the "rate" (how
+much of the common state buffer gets updated with new data between invocations
+of the Keccak function and analogous to the "block size"), what domain
+separation suffix/padding gets appended to the message and how much data is
+extracted at the end. The Keccak sponge function is designed such that
+arbitrary amounts of output can be obtained for certain algorithms.
+
+Four standard digest algorithms are provided:
+
+ - SHA3-224
+ - SHA3-256
+ - SHA3-384
+ - SHA3-512
+
+and two Extendable-Output Functions (XOF):
+
+ - SHAKE128
+ - SHAKE256
+
+If selectable algorithms are required then the crypto_hash API may be used
+instead as this binds each algorithm to a specific C type.
+
+
+Basic API
+=========
+
+The basic API has a separate context struct for each algorithm in the SHA3
+suite, none of the contents of which are expected to be accessed directly::
+
+ struct sha3_224_ctx { ... };
+ struct sha3_256_ctx { ... };
+ struct sha3_384_ctx { ... };
+ struct sha3_512_ctx { ... };
+ struct shake128_ctx { ... };
+ struct shake256_ctx { ... };
+
+There are a collection of initialisation functions, one for each algorithm
+supported, that initialise the context appropriately for that algorithm::
+
+ void sha3_224_init(struct sha3_224_ctx *ctx);
+ void sha3_256_init(struct sha3_256_ctx *ctx);
+ void sha3_384_init(struct sha3_384_ctx *ctx);
+ void sha3_512_init(struct sha3_512_ctx *ctx);
+ void shake128_init(struct shake128_ctx *ctx);
+ void shake256_init(struct shake256_ctx *ctx);
+
+Data is then added with the appropriate update function, again one per
+algorithm::
+
+ void sha3_224_update(struct sha3_224_ctx *ctx,
+ const u8 *data, size_t len);
+ void sha3_256_update(struct sha3_256_ctx *ctx,
+ const u8 *data, size_t len);
+ void sha3_384_update(struct sha3_384_ctx *ctx,
+ const u8 *data, size_t len);
+ void sha3_512_update(struct sha3_512_ctx *ctx,
+ const u8 *data, size_t len);
+ void shake128_update(struct shake128_ctx *ctx,
+ const u8 *data, size_t len);
+ void shake256_update(struct shake256_ctx *ctx,
+ const u8 *data, size_t len);
+
+The update function may be called multiple times if need be to add
+non-contiguous data.
+
+For digest algorithms, the digest is finalised and extracted with the
+algorithm-specific function::
+
+ void sha3_224_final(struct sha3_224_ctx *ctx,
+ u8 out[SHA3_224_DIGEST_SIZE]);
+ void sha3_256_final(struct sha3_256_ctx *ctx,
+ u8 out[SHA3_256_DIGEST_SIZE]);
+ void sha3_384_final(struct sha3_384_ctx *ctx,
+ u8 out[SHA3_384_DIGEST_SIZE]);
+ void sha3_512_final(struct sha3_512_ctx *ctx,
+ u8 out[SHA3_512_DIGEST_SIZE]);
+
+which also explicitly clears the context. The amount of data extracted is
+determined by the type.
+
+
+Extendable-Output Functions
+---------------------------
+
+For XOFs, once the data has been added to a context, a variable amount of data
+may be extracted. This can be done by calling the appropriate squeeze
+function::
+
+ void shake128_squeeze(struct shake128_ctx *ctx, u8 *out, size_t out_len);
+ void shake256_squeeze(struct shake256_ctx *ctx, u8 *out, size_t out_len);
+
+and telling it how much data should be extracted. The squeeze function may be
+called multiple times but it will only append the domain separation suffix on
+the first invocation.
+
+Note that performing a number of squeezes, with the output laid consequitively
+in a buffer, gets exactly the same output as doing a single squeeze for the
+combined amount over the same buffer.
+
+Once all the desired output has been extracted, the context should be cleared
+with the clear function appropriate to the algorithm::
+
+ void shake128_clear(struct shake128_ctx *ctx);
+ void shake256_clear(struct shake256_ctx *ctx);
+
+
+Convenience API
+===============
+
+It only a single contiguous buffer of input needs to be added and only a single
+buffer of digest or XOF output is required, then a convenience API is provided
+that wraps all the required steps into a single function. There is one
+function for each algorithm supported::
+
+ void sha3_224(const u8 *in, size_t in_len, u8 out[SHA3_224_DIGEST_SIZE]);
+ void sha3_256(const u8 *in, size_t in_len, u8 out[SHA3_256_DIGEST_SIZE]);
+ void sha3_384(const u8 *in, size_t in_len, u8 out[SHA3_384_DIGEST_SIZE]);
+ void sha3_512(const u8 *in, size_t in_len, u8 out[SHA3_512_DIGEST_SIZE]);
+ void shake128(const u8 *in, size_t in_len, u8 *out, size_t out_len);
+ void shake256(const u8 *in, size_t in_len, u8 *out, size_t out_len);
+
+
+Internal API
+============
+
+There is a common internal API underlying all of this that may be used to build
+further algorithms or APIs as the engine in the same in all cases. The
+algorithm APIs all wrap the common context structure::
+
+ struct sha3_ctx {
+ struct sha3_state state;
+ u8 block_size;
+ u8 padding;
+ u8 absorb_offset;
+ u8 squeeze_offset;
+ bool end_marked;
+ };
+
+ struct sha3_state {
+ u64 st[SHA3_STATE_SIZE / 8];
+ };
+
+The fields are as follows:
+
+ * ``state.st``
+
+ An array of 25 64-bit state buckets that are used to hold the mathematical
+ state of the Keccak engine. Data is XOR'd onto part of this, the engine is
+ cranked and then the output is copied from this.
+
+ For the convenience of adding input and extract output from it, the array is
+ kept in little-endian order most of the time, but is byteswapped to
+ host-endian in order to perform the Keccak function and then byteswapped
+ back again. On an LE machine, the byteswapping is a no-op.
+
+ * ``block_size``
+
+ The size of the block of state that can be updated or extracted at a time.
+ This is related to the algorithm size and is analogous to the "rate" in the
+ algorithm definition.
+
+ * ``padding``
+
+ The terminating byte to add when finalising the stat. This may differ
+ between algorithms.
+
+ * ``absorb_offset``
+
+ This tracks which is the next byte of state to be updated; when it hits
+ ``block_size``, the engine is cranked and this is reset to 0.
+
+ * ``squeeze_offset``
+
+ This tracks which is the next byte of state to be extracted; similar to
+ ``partial``, when it hits ``block_size``, if more output is requested, the
+ engine is cranked to generate more and this is reset to 0.
+
+ * ``end_marked``
+
+ This is set to true when the domain separation suffix and any padding have
+ been appended to the state to prevent multiple squeezings from XOF
+ algorithms from re-appending this.
+
+Note that the size of the digest is *not* included here as that's only needed
+at finalisation time for digest algorithms and can be supplied then. It is not
+relevant to XOFs.
+
+To make use of the context, the following internal functions are provided::
+
+ void sha3_update(struct sha3_ctx *ctx, const u8 *data, size_t len);
+ void sha3_squeeze(struct sha3_ctx *ctx, u8 *out, size_t out_len);
+ void sha3_clear(struct sha3_ctx *ctx);
+
+These allow data to be appended to/absorbed into the state, output to be
+extracted/squeezed from the state and for the state to be cleared. Note that
+there is no "final" function, per se, but that can be constructed by squeezing
+and clearing.
+
+
+Testing
+-------
+
+The sha3 module does a basic sanity test on initialisation, but there is also a
+kunit test module available.
+
+
+References
+==========
+
+[1] https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.202.pdf
+
+
+
+API Function Reference
+======================
+
+.. kernel-doc:: crypto/lib/sha3.c
+.. kernel-doc:: include/crypto/sha3.h
diff --git a/include/crypto/sha3.h b/include/crypto/sha3.h
index 41e1b83a6d91..a54117cb1546 100644
--- a/include/crypto/sha3.h
+++ b/include/crypto/sha3.h
@@ -1,11 +1,14 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Common values for SHA-3 algorithms
+ *
+ * See also Documentation/crypto/sha3.rst
*/
#ifndef __CRYPTO_SHA3_H__
#define __CRYPTO_SHA3_H__
#include <linux/types.h>
+#include <linux/string.h>
#define SHA3_224_DIGEST_SIZE (224 / 8)
#define SHA3_224_BLOCK_SIZE (200 - 2 * SHA3_224_DIGEST_SIZE)
@@ -23,6 +26,14 @@
#define SHA3_512_BLOCK_SIZE (200 - 2 * SHA3_512_DIGEST_SIZE)
#define SHA3_512_EXPORT_SIZE SHA3_STATE_SIZE + SHA3_512_BLOCK_SIZE + 1
+/* SHAKE128 and SHAKE256 actually have variable output size, but this is
+ * used to calculate the block size/rate analogously to the above..
+ */
+#define SHAKE128_DEFAULT_SIZE (128 / 8)
+#define SHAKE128_BLOCK_SIZE (200 - 2 * SHAKE128_DEFAULT_SIZE)
+#define SHAKE256_DEFAULT_SIZE (256 / 8)
+#define SHAKE256_BLOCK_SIZE (200 - 2 * SHAKE256_DEFAULT_SIZE)
+
#define SHA3_STATE_SIZE 200
struct shash_desc;
@@ -31,6 +42,426 @@ struct sha3_state {
u64 st[SHA3_STATE_SIZE / 8];
};
+/*
+ * The SHA3 context structure and state buffer.
+ *
+ * To avoid the need to byteswap when adding input and extracting output from
+ * the state array, the state array is kept in little-endian order most of the
+ * time, but is byteswapped to host-endian in order to perform the Keccak
+ * function and then byteswapped back again after. On a LE machine, the
+ * byteswap step is a no-op.
+ */
+struct sha3_ctx {
+ struct sha3_state state;
+ u8 block_size; /* Block size in bytes */
+ u8 padding; /* Padding byte */
+ u8 absorb_offset; /* Next state byte to absorb into */
+ u8 squeeze_offset; /* Next state byte to extract */
+ bool end_marked; /* T if end marker inserted */
+};
+
int crypto_sha3_init(struct shash_desc *desc);
-#endif
+/**
+ * sha3_clear() - Explicitly clear the entire context
+ * @ctx: the context to clear
+ *
+ * Explicitly clear the entire context, including the type parameters; after
+ * this, the context must be fully initialized again.
+ *
+ * Context: Any context.
+ */
+static inline void sha3_clear(struct sha3_ctx *ctx)
+{
+ memzero_explicit(ctx, sizeof(*ctx));
+}
+
+void sha3_update(struct sha3_ctx *ctx, const u8 *data, size_t len);
+void sha3_squeeze(struct sha3_ctx *ctx, u8 *out, size_t out_len);
+
+/*
+ * Context wrapper for SHA3-224.
+ */
+struct sha3_224_ctx {
+ struct sha3_ctx ctx;
+};
+
+/**
+ * sha3_224_init() - Set a SHA3 context for SHA3-224
+ * @ctx: the context to initialize
+ *
+ * Initialize a SHA3 context for the production of a SHA3-224 digest of a
+ * message.
+ *
+ * Context: Any context.
+ */
+static inline void sha3_224_init(struct sha3_224_ctx *ctx)
+{
+ *ctx = (struct sha3_224_ctx){
+ .ctx.block_size = SHA3_224_BLOCK_SIZE,
+ .ctx.padding = 0x06,
+ };
+}
+
+/**
+ * sha3_224_update() - Update a SHA3-224 hash with message data
+ * @ctx: the context to update; must have been initialized
+ * @data: the message data
+ * @len: the data length in bytes
+ *
+ * This can be called any number of times to add data to the hash, performing
+ * the "keccak sponge absorbing" phase.
+ *
+ * Context: Any context.
+ */
+static inline void sha3_224_update(struct sha3_224_ctx *ctx, const u8 *data, size_t len)
+{
+ return sha3_update(&ctx->ctx, data, len);
+}
+
+/**
+ * sha3_224_final() - Finalise a SHA3-224 hash and extract the digest
+ * @ctx: The context to finalise; must have been initialized
+ * @out: Where to write the resulting message digest
+ *
+ * Finish the computation of a SHA3-224 hash and perform the "Keccak sponge
+ * squeezing" phase. The digest is written to @out buffer and the context will
+ * be completely zeroed out.
+ *
+ * Context: Any context.
+ */
+static inline void sha3_224_final(struct sha3_224_ctx *ctx, u8 out[SHA3_224_DIGEST_SIZE])
+{
+ sha3_squeeze(&ctx->ctx, out, SHA3_224_DIGEST_SIZE);
+ sha3_clear(&ctx->ctx);
+}
+
+/*
+ * Context wrapper for SHA3-256.
+ */
+struct sha3_256_ctx {
+ struct sha3_ctx ctx;
+};
+
+/**
+ * sha3_256_init() - Set a SHA3 context for SHA3-256
+ * @ctx: the context to initialize
+ *
+ * Initialize a SHA3 context for the production of a SHA3-256 digest of a
+ * message.
+ *
+ * Context: Any context.
+ */
+static inline void sha3_256_init(struct sha3_256_ctx *ctx)
+{
+ *ctx = (struct sha3_256_ctx){
+ .ctx.block_size = SHA3_256_BLOCK_SIZE,
+ .ctx.padding = 0x06,
+ };
+}
+
+/**
+ * sha3_256_update() - Update a SHA3-256 hash with message data
+ * @ctx: the context to update; must have been initialized
+ * @data: the message data
+ * @len: the data length in bytes
+ *
+ * This can be called any number of times to add data to the hash, performing
+ * the "keccak sponge absorbing" phase.
+ *
+ * Context: Any context.
+ */
+static inline void sha3_256_update(struct sha3_256_ctx *ctx, const u8 *data, size_t len)
+{
+ return sha3_update(&ctx->ctx, data, len);
+}
+
+/**
+ * sha3_256_final() - Finalise a SHA3-256 hash and extract the digest
+ * @ctx: The context to finalise; must have been initialized
+ * @out: Where to write the resulting message digest
+ *
+ * Finish the computation of a SHA3-256 hash and perform the "Keccak sponge
+ * squeezing" phase. The digest is written to @out buffer and the context will
+ * be completely zeroed out.
+ *
+ * Context: Any context.
+ */
+static inline void sha3_256_final(struct sha3_256_ctx *ctx, u8 out[SHA3_256_DIGEST_SIZE])
+{
+ sha3_squeeze(&ctx->ctx, out, SHA3_256_DIGEST_SIZE);
+ sha3_clear(&ctx->ctx);
+}
+
+/*
+ * Context wrapper for SHA3-384.
+ */
+struct sha3_384_ctx {
+ struct sha3_ctx ctx;
+};
+
+/**
+ * sha3_384_init() - Set a SHA3 context for SHA3-384
+ * @ctx: the context to initialize
+ *
+ * Initialize a SHA3 context for the production of a SHA3-384 digest of a
+ * message.
+ *
+ * Context: Any context.
+ */
+static inline void sha3_384_init(struct sha3_384_ctx *ctx)
+{
+ *ctx = (struct sha3_384_ctx){
+ .ctx.block_size = SHA3_384_BLOCK_SIZE,
+ .ctx.padding = 0x06,
+ };
+}
+
+/**
+ * sha3_384_update() - Update a SHA3-384 hash with message data
+ * @ctx: the context to update; must have been initialized
+ * @data: the message data
+ * @len: the data length in bytes
+ *
+ * This can be called any number of times to add data to the hash, performing
+ * the "keccak sponge absorbing" phase.
+ *
+ * Context: Any context.
+ */
+static inline void sha3_384_update(struct sha3_384_ctx *ctx, const u8 *data, size_t len)
+{
+ return sha3_update(&ctx->ctx, data, len);
+}
+
+/**
+ * sha3_384_final() - Finalise a SHA3-384 hash and extract the digest
+ * @ctx: The context to finalise; must have been initialized
+ * @out: Where to write the resulting message digest
+ *
+ * Finish the computation of a SHA3-384 hash and perform the "Keccak sponge
+ * squeezing" phase. The digest is written to @out buffer and the context will
+ * be completely zeroed out.
+ *
+ * Context: Any context.
+ */
+static inline void sha3_384_final(struct sha3_384_ctx *ctx, u8 out[SHA3_384_DIGEST_SIZE])
+{
+ sha3_squeeze(&ctx->ctx, out, SHA3_384_DIGEST_SIZE);
+ sha3_clear(&ctx->ctx);
+}
+
+/*
+ * Context wrapper for SHA3-512.
+ */
+struct sha3_512_ctx {
+ struct sha3_ctx ctx;
+};
+
+/**
+ * sha3_512_init() - Set a SHA3 context for SHA3-512
+ * @ctx: the context to initialize
+ *
+ * Initialize a SHA3 context for the production of a SHA3-512 digest of a
+ * message.
+ *
+ * Context: Any context.
+ */
+static inline void sha3_512_init(struct sha3_512_ctx *ctx)
+{
+ *ctx = (struct sha3_512_ctx){
+ .ctx.block_size = SHA3_512_BLOCK_SIZE,
+ .ctx.padding = 0x06,
+ };
+}
+
+/**
+ * sha3_512_update() - Update a SHA3-512 hash with message data
+ * @ctx: the context to update; must have been initialized
+ * @data: the message data
+ * @len: the data length in bytes
+ *
+ * This can be called any number of times to add data to the hash, performing
+ * the "keccak sponge absorbing" phase.
+ *
+ * Context: Any context.
+ */
+static inline void sha3_512_update(struct sha3_512_ctx *ctx, const u8 *data, size_t len)
+{
+ return sha3_update(&ctx->ctx, data, len);
+}
+
+/**
+ * sha3_512_final() - Finalise a SHA3-512 hash and extract the digest
+ * @ctx: The context to finalise; must have been initialized
+ * @out: Where to write the resulting message digest
+ *
+ * Finish the computation of a SHA3-512 hash and perform the "Keccak sponge
+ * squeezing" phase. The digest is written to @out buffer and the context will
+ * be completely zeroed out.
+ *
+ * Context: Any context.
+ */
+static inline void sha3_512_final(struct sha3_512_ctx *ctx, u8 out[SHA3_512_DIGEST_SIZE])
+{
+ sha3_squeeze(&ctx->ctx, out, SHA3_512_DIGEST_SIZE);
+ sha3_clear(&ctx->ctx);
+}
+
+/*
+ * Context wrapper for SHAKE128.
+ */
+struct shake128_ctx {
+ struct sha3_ctx ctx;
+};
+
+/**
+ * shake128_init() - Set a SHA3 context for SHAKE128
+ * @ctx: The context to initialize
+ *
+ * Initialize a SHA3 context for the production of SHAKE128 output generation
+ * from a message. The sha3_squeeze() function can be used to extract an
+ * arbitrary amount of data from the context.
+ *
+ * Context: Any context.
+ */
+static inline void shake128_init(struct shake128_ctx *ctx)
+{
+ *ctx = (struct shake128_ctx){
+ .ctx.block_size = SHAKE128_BLOCK_SIZE,
+ .ctx.padding = 0x1f,
+ };
+}
+
+/**
+ * shake128_update() - Update a SHAKE128 hash with message data
+ * @ctx: the context to update; must have been initialized
+ * @data: the message data
+ * @len: the data length in bytes
+ *
+ * This can be called any number of times to add data to the XOF state,
+ * performing the "keccak sponge absorbing" phase.
+ *
+ * Context: Any context.
+ */
+static inline void shake128_update(struct shake128_ctx *ctx, const u8 *data, size_t len)
+{
+ return sha3_update(&ctx->ctx, data, len);
+}
+
+/**
+ * shake128_squeeze() - Finalize a SHAKE128 digest of any type and extract the digest
+ * @ctx: the context to finalize; must have been initialized
+ * @out: Where to write the resulting message digest
+ * @out_size: The amount of digest to extract to @out in bytes
+ *
+ * Finish the computation of a SHAKE128 XOF and perform the "Keccak sponge
+ * squeezing" phase. @out_size amount of digest is written to @out buffer.
+ *
+ * This may be called multiple times to extract continuations of the digest.
+ * Note that, a number of consecutive squeezes laid end-to-end will yield the
+ * same output as one big squeeze generating the same total amount of output.
+ *
+ * Context: Any context.
+ */
+static inline void shake128_squeeze(struct shake128_ctx *ctx, u8 *out, size_t out_size)
+{
+ return sha3_squeeze(&ctx->ctx, out, out_size);
+}
+
+/**
+ * shake128_clear() - Explicitly clear the entire SHAKE128 context
+ * @ctx: the context to clear
+ *
+ * Explicitly clear the entire context; after this, the context must be
+ * initialized again.
+ *
+ * Context: Any context.
+ */
+static inline void shake128_clear(struct shake128_ctx *ctx)
+{
+ sha3_clear(&ctx->ctx);
+}
+
+/*
+ * Context wrapper for SHAKE256.
+ */
+struct shake256_ctx {
+ struct sha3_ctx ctx;
+};
+
+/**
+ * shake256_init() - Set a SHA3 context for SHAKE256
+ * @ctx: The context to initialize
+ *
+ * Initialize a SHA3 context for the production of SHAKE256 output generation
+ * from a message. The sha3_squeeze() function can be used to extract an
+ * arbitrary amount of data from the context.
+ *
+ * Context: Any context.
+ */
+static inline void shake256_init(struct shake256_ctx *ctx)
+{
+ *ctx = (struct shake256_ctx){
+ .ctx.block_size = SHAKE256_BLOCK_SIZE,
+ .ctx.padding = 0x1f,
+ };
+}
+
+/**
+ * shake256_update() - Update a SHAKE256 hash with message data
+ * @ctx: the context to update; must have been initialized
+ * @data: the message data
+ * @len: the data length in bytes
+ *
+ * This can be called any number of times to add data to the XOF state,
+ * performing the "keccak sponge absorbing" phase.
+ *
+ * Context: Any context.
+ */
+static inline void shake256_update(struct shake256_ctx *ctx, const u8 *data, size_t len)
+{
+ return sha3_update(&ctx->ctx, data, len);
+}
+
+/**
+ * shake256_squeeze() - Finalize a SHAKE256 digest of any type and extract the digest
+ * @ctx: the context to finalize; must have been initialized
+ * @out: Where to write the resulting message digest
+ * @out_size: The amount of digest to extract to @out in bytes
+ *
+ * Finish the computation of a SHAKE256 XOF and perform the "Keccak sponge
+ * squeezing" phase. @out_size amount of digest is written to @out buffer.
+ *
+ * This may be called multiple times to extract continuations of the digest.
+ * Note that, a number of consecutive squeezes laid end-to-end will yield the
+ * same output as one big squeeze generating the same total amount of output.
+ *
+ * Context: Any context.
+ */
+static inline void shake256_squeeze(struct shake256_ctx *ctx, u8 *out, size_t out_size)
+{
+ return sha3_squeeze(&ctx->ctx, out, out_size);
+}
+
+/**
+ * shake256_clear() - Explicitly clear the entire SHAKE256 context
+ * @ctx: the context to clear
+ *
+ * Explicitly clear the entire context; after this, the context must be
+ * initialized again.
+ *
+ * Context: Any context.
+ */
+static inline void shake256_clear(struct shake256_ctx *ctx)
+{
+ sha3_clear(&ctx->ctx);
+}
+
+void sha3_224(const u8 *in, size_t in_len, u8 out[SHA3_224_DIGEST_SIZE]);
+void sha3_256(const u8 *in, size_t in_len, u8 out[SHA3_256_DIGEST_SIZE]);
+void sha3_384(const u8 *in, size_t in_len, u8 out[SHA3_384_DIGEST_SIZE]);
+void sha3_512(const u8 *in, size_t in_len, u8 out[SHA3_512_DIGEST_SIZE]);
+void shake128(const u8 *in, size_t in_len, u8 *out, size_t out_len);
+void shake256(const u8 *in, size_t in_len, u8 *out, size_t out_len);
+
+#endif /* __CRYPTO_SHA3_H__ */
diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig
index eea17e36a22b..1caf3fbe24bf 100644
--- a/lib/crypto/Kconfig
+++ b/lib/crypto/Kconfig
@@ -184,6 +184,13 @@ config CRYPTO_LIB_SHA512_ARCH
default y if SPARC64
default y if X86_64
+config CRYPTO_LIB_SHA3
+ tristate
+ help
+ The SHA3 library functions.
+ Select this if your module uses any of these functions from
+ <crypto/sha3.h>.
+
config CRYPTO_LIB_SM3
tristate
diff --git a/lib/crypto/Makefile b/lib/crypto/Makefile
index bded351aeace..b7fa51bfeebb 100644
--- a/lib/crypto/Makefile
+++ b/lib/crypto/Makefile
@@ -268,6 +268,12 @@ endif # CONFIG_CRYPTO_LIB_SHA512_ARCH
################################################################################
+#obj-$(CONFIG_CRYPTO_LIB_SHA3) += libsha3.o
+#libsha3-y := sha3.o
+obj-$(CONFIG_CRYPTO_SHA3) += sha3.o
+
+################################################################################
+
obj-$(CONFIG_MPILIB) += mpi/
obj-$(CONFIG_CRYPTO_SELFTESTS_FULL) += simd.o
diff --git a/lib/crypto/sha3.c b/lib/crypto/sha3.c
new file mode 100644
index 000000000000..5f847a5eecdc
--- /dev/null
+++ b/lib/crypto/sha3.c
@@ -0,0 +1,511 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Cryptographic API.
+ *
+ * SHA-3, as specified in
+ * https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.202.pdf
+ *
+ * SHA-3 code by Jeff Garzik <jeff@garzik.org>
+ * Ard Biesheuvel <ard.biesheuvel@linaro.org>
+ * David Howells <dhowells@redhat.com>
+ *
+ * See also Documentation/crypto/sha3.rst
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <crypto/sha3.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/unaligned.h>
+
+/*
+ * On some 32-bit architectures, such as h8300, GCC ends up using over 1 KB of
+ * stack if the round calculation gets inlined into the loop in
+ * sha3_keccakf_rounds_generic(). On the other hand, on 64-bit architectures
+ * with plenty of [64-bit wide] general purpose registers, not inlining it
+ * severely hurts performance. So let's use 64-bitness as a heuristic to
+ * decide whether to inline or not.
+ */
+#ifdef CONFIG_64BIT
+#define SHA3_INLINE inline
+#else
+#define SHA3_INLINE noinline
+#endif
+
+#define SHA3_KECCAK_ROUNDS 24
+
+static const u64 sha3_keccakf_rndc[24] = {
+ 0x0000000000000001ULL, 0x0000000000008082ULL, 0x800000000000808aULL,
+ 0x8000000080008000ULL, 0x000000000000808bULL, 0x0000000080000001ULL,
+ 0x8000000080008081ULL, 0x8000000000008009ULL, 0x000000000000008aULL,
+ 0x0000000000000088ULL, 0x0000000080008009ULL, 0x000000008000000aULL,
+ 0x000000008000808bULL, 0x800000000000008bULL, 0x8000000000008089ULL,
+ 0x8000000000008003ULL, 0x8000000000008002ULL, 0x8000000000000080ULL,
+ 0x000000000000800aULL, 0x800000008000000aULL, 0x8000000080008081ULL,
+ 0x8000000000008080ULL, 0x0000000080000001ULL, 0x8000000080008008ULL
+};
+
+/*
+ * Perform a single round of Keccak mixing.
+ */
+static SHA3_INLINE void sha3_keccakf_one_round_generic(struct sha3_state *state)
+{
+ u64 *st = state->st;
+ u64 t[5], tt, bc[5];
+
+ /* Theta */
+ bc[0] = st[0] ^ st[5] ^ st[10] ^ st[15] ^ st[20];
+ bc[1] = st[1] ^ st[6] ^ st[11] ^ st[16] ^ st[21];
+ bc[2] = st[2] ^ st[7] ^ st[12] ^ st[17] ^ st[22];
+ bc[3] = st[3] ^ st[8] ^ st[13] ^ st[18] ^ st[23];
+ bc[4] = st[4] ^ st[9] ^ st[14] ^ st[19] ^ st[24];
+
+ t[0] = bc[4] ^ rol64(bc[1], 1);
+ t[1] = bc[0] ^ rol64(bc[2], 1);
+ t[2] = bc[1] ^ rol64(bc[3], 1);
+ t[3] = bc[2] ^ rol64(bc[4], 1);
+ t[4] = bc[3] ^ rol64(bc[0], 1);
+
+ st[0] ^= t[0];
+
+ /* Rho Pi */
+ tt = st[1];
+ st[ 1] = rol64(st[ 6] ^ t[1], 44);
+ st[ 6] = rol64(st[ 9] ^ t[4], 20);
+ st[ 9] = rol64(st[22] ^ t[2], 61);
+ st[22] = rol64(st[14] ^ t[4], 39);
+ st[14] = rol64(st[20] ^ t[0], 18);
+ st[20] = rol64(st[ 2] ^ t[2], 62);
+ st[ 2] = rol64(st[12] ^ t[2], 43);
+ st[12] = rol64(st[13] ^ t[3], 25);
+ st[13] = rol64(st[19] ^ t[4], 8);
+ st[19] = rol64(st[23] ^ t[3], 56);
+ st[23] = rol64(st[15] ^ t[0], 41);
+ st[15] = rol64(st[ 4] ^ t[4], 27);
+ st[ 4] = rol64(st[24] ^ t[4], 14);
+ st[24] = rol64(st[21] ^ t[1], 2);
+ st[21] = rol64(st[ 8] ^ t[3], 55);
+ st[ 8] = rol64(st[16] ^ t[1], 45);
+ st[16] = rol64(st[ 5] ^ t[0], 36);
+ st[ 5] = rol64(st[ 3] ^ t[3], 28);
+ st[ 3] = rol64(st[18] ^ t[3], 21);
+ st[18] = rol64(st[17] ^ t[2], 15);
+ st[17] = rol64(st[11] ^ t[1], 10);
+ st[11] = rol64(st[ 7] ^ t[2], 6);
+ st[ 7] = rol64(st[10] ^ t[0], 3);
+ st[10] = rol64( tt ^ t[1], 1);
+
+ /* Chi */
+ bc[ 0] = ~st[ 1] & st[ 2];
+ bc[ 1] = ~st[ 2] & st[ 3];
+ bc[ 2] = ~st[ 3] & st[ 4];
+ bc[ 3] = ~st[ 4] & st[ 0];
+ bc[ 4] = ~st[ 0] & st[ 1];
+ st[ 0] ^= bc[ 0];
+ st[ 1] ^= bc[ 1];
+ st[ 2] ^= bc[ 2];
+ st[ 3] ^= bc[ 3];
+ st[ 4] ^= bc[ 4];
+
+ bc[ 0] = ~st[ 6] & st[ 7];
+ bc[ 1] = ~st[ 7] & st[ 8];
+ bc[ 2] = ~st[ 8] & st[ 9];
+ bc[ 3] = ~st[ 9] & st[ 5];
+ bc[ 4] = ~st[ 5] & st[ 6];
+ st[ 5] ^= bc[ 0];
+ st[ 6] ^= bc[ 1];
+ st[ 7] ^= bc[ 2];
+ st[ 8] ^= bc[ 3];
+ st[ 9] ^= bc[ 4];
+
+ bc[ 0] = ~st[11] & st[12];
+ bc[ 1] = ~st[12] & st[13];
+ bc[ 2] = ~st[13] & st[14];
+ bc[ 3] = ~st[14] & st[10];
+ bc[ 4] = ~st[10] & st[11];
+ st[10] ^= bc[ 0];
+ st[11] ^= bc[ 1];
+ st[12] ^= bc[ 2];
+ st[13] ^= bc[ 3];
+ st[14] ^= bc[ 4];
+
+ bc[ 0] = ~st[16] & st[17];
+ bc[ 1] = ~st[17] & st[18];
+ bc[ 2] = ~st[18] & st[19];
+ bc[ 3] = ~st[19] & st[15];
+ bc[ 4] = ~st[15] & st[16];
+ st[15] ^= bc[ 0];
+ st[16] ^= bc[ 1];
+ st[17] ^= bc[ 2];
+ st[18] ^= bc[ 3];
+ st[19] ^= bc[ 4];
+
+ bc[ 0] = ~st[21] & st[22];
+ bc[ 1] = ~st[22] & st[23];
+ bc[ 2] = ~st[23] & st[24];
+ bc[ 3] = ~st[24] & st[20];
+ bc[ 4] = ~st[20] & st[21];
+ st[20] ^= bc[ 0];
+ st[21] ^= bc[ 1];
+ st[22] ^= bc[ 2];
+ st[23] ^= bc[ 3];
+ st[24] ^= bc[ 4];
+}
+
+static void sha3_keccakf_rounds_generic(struct sha3_state *state)
+{
+ for (int round = 0; round < SHA3_KECCAK_ROUNDS; round++) {
+ sha3_keccakf_one_round_generic(state);
+ /* Iota */
+ state->st[0] ^= sha3_keccakf_rndc[round];
+ }
+}
+
+/*
+ * Byteswap the state buckets to CPU-endian if we're not on a little-endian
+ * machine for the duration of the Keccak mixing function. Note that these
+ * loops are no-ops on LE machines and will be optimised away.
+ */
+static void sha3_keccakf_generic(struct sha3_state *state)
+{
+ for (int i = 0; i < ARRAY_SIZE(state->st); i++)
+ cpu_to_le64s(&state->st[i]);
+
+ sha3_keccakf_rounds_generic(state);
+
+ for (int i = 0; i < ARRAY_SIZE(state->st); i++)
+ le64_to_cpus(&state->st[i]);
+}
+
+static void sha3_absorb_block_generic(struct sha3_ctx *ctx, const u8 *data)
+{
+ struct sha3_state *state = &ctx->state;
+ size_t bsize = ctx->block_size;
+
+ for (size_t i = 0; i < bsize / 8; i++)
+ state->st[i] ^= get_unaligned((u64 *)(data + 8 * i));
+ sha3_keccakf_generic(state);
+}
+
+/*
+ * Perform rounds of XOR'ing whole blocks of data into the state buffer and
+ * then performing a keccak mix step.
+ */
+static void sha3_absorb_blocks_generic(struct sha3_ctx *ctx,
+ const u8 *data, size_t nblocks)
+{
+ do {
+ sha3_absorb_block_generic(ctx, data);
+ data += ctx->block_size;
+ } while (--nblocks);
+}
+
+#ifdef CONFIG_CRYPTO_LIB_SHA3_ARCH
+#include "sha3.h" /* $(SRCARCH)/sha3.h */
+#else
+#define sha3_keccakf sha3_keccakf_generic
+#define sha3_absorb_blocks sha3_absorb_blocks_generic
+#endif
+
+/*
+ * XOR in partial data that's insufficient to fill a whole block.
+ */
+static void sha3_absorb_xorle(struct sha3_ctx *ctx, const u8 *data,
+ size_t partial, size_t len)
+{
+ u8 *buf = (u8 *)ctx->state.st;
+
+ buf += partial;
+ for (size_t i = 0; i < len; i++)
+ *buf++ ^= *data++;
+}
+
+/**
+ * sha3_update() - Update a SHA3 context of any type with message data
+ * @ctx: the context to update; must have been initialized
+ * @data: the message data
+ * @len: the data length in bytes
+ *
+ * This can be called any number of times to perform the "keccak sponge
+ * absorbing" phase.
+ *
+ * Context: May use the FPU/Vector unit registers.
+ */
+void sha3_update(struct sha3_ctx *ctx, const u8 *data, size_t len)
+{
+ size_t absorb_offset = ctx->absorb_offset;
+ size_t bsize = ctx->block_size;
+
+ WARN_ON_ONCE(ctx->end_marked);
+
+ if (absorb_offset && absorb_offset + len >= bsize) {
+ sha3_absorb_xorle(ctx, data, absorb_offset, bsize - absorb_offset);
+ len -= bsize - absorb_offset;
+ data += bsize - absorb_offset;
+ sha3_keccakf(&ctx->state);
+ ctx->absorb_offset = 0;
+ }
+
+ if (len >= bsize) {
+ size_t nblocks = len / bsize;
+
+ sha3_absorb_blocks(ctx, data, nblocks);
+ data += nblocks * bsize;
+ len -= nblocks * bsize;
+ }
+
+ if (len) {
+ sha3_absorb_xorle(ctx, data, ctx->absorb_offset, len);
+ ctx->absorb_offset += len;
+ }
+}
+EXPORT_SYMBOL_GPL(sha3_update);
+
+/**
+ * sha3_squeeze() - Finalize a SHA3 digest of any type and extract the digest
+ * @ctx: the context to finalize; must have been initialized
+ * @out: Where to write the resulting message digest
+ * @out_size: The amount of digest to extract to @out
+ *
+ * Finish the computation of a SHA3 message digest of any type and perform the
+ * "Keccak sponge squeezing" phase. @out_size amount of digest is written to
+ * @out buffer.
+ *
+ * This may be called multiple times to extract continuations of the digest.
+ * Note that, for example, two consecutive 16-byte squeezes laid end-to-end
+ * will yield the same as one 32-byte squeeze.
+ *
+ * Context: May use the FPU/Vector unit registers.
+ */
+void sha3_squeeze(struct sha3_ctx *ctx, u8 *out, size_t out_size)
+{
+ size_t squeeze_offset = ctx->squeeze_offset;
+ size_t bsize = ctx->block_size;
+ u8 *p = (u8 *)ctx->state.st, end_marker = 0x80;
+
+ if (!ctx->end_marked) {
+ sha3_absorb_xorle(ctx, &ctx->padding, ctx->absorb_offset, 1);
+ sha3_absorb_xorle(ctx, &end_marker, bsize - 1, 1);
+ ctx->end_marked = true;
+ }
+
+ for (;;) {
+ if (squeeze_offset == 0)
+ sha3_keccakf(&ctx->state);
+
+ size_t part = umin(out_size, bsize - squeeze_offset);
+
+ if (part > 0) {
+ memcpy(out, p + squeeze_offset, part);
+ out_size -= part;
+ out += part;
+ squeeze_offset += part;
+ }
+ if (!out_size)
+ break;
+ if (squeeze_offset >= bsize)
+ squeeze_offset = 0;
+ }
+
+ ctx->squeeze_offset = squeeze_offset;
+}
+EXPORT_SYMBOL_GPL(sha3_squeeze);
+
+/**
+ * sha3_224() - Convenience wrapper to digest a simple buffer as SHA3-224
+ * @in: The data to be digested
+ * @in_len: The amount of data to be digested in bytes
+ * @out: The buffer into which the digest will be stored (size not checked)
+ *
+ * Convenience wrapper to initialise a SHA3 context for SHA-224, add the input
+ * data to it, finalise it, extract 28 bytes of digest and clear the context.
+ *
+ * Context: May use the FPU/Vector unit registers.
+ */
+void sha3_224(const u8 *in, size_t in_len, u8 out[SHA3_224_DIGEST_SIZE])
+{
+ struct sha3_224_ctx ctx;
+
+ sha3_224_init(&ctx);
+ sha3_224_update(&ctx, in, in_len);
+ sha3_224_final(&ctx, out);
+}
+EXPORT_SYMBOL(sha3_224);
+
+/**
+ * sha3_256() - Convenience wrapper to digest a simple buffer as SHA3-256
+ * @in: The data to be digested
+ * @in_len: The amount of data to be digested in bytes
+ * @out: The buffer into which the digest will be stored (size not checked)
+ *
+ * Convenience wrapper to initialise a SHA3 context for SHA-256, add the input
+ * data to it, finalise it, extract 32 bytes of digest and clear the context.
+ *
+ * Context: May use the FPU/Vector unit registers.
+ */
+void sha3_256(const u8 *in, size_t in_len, u8 out[SHA3_256_DIGEST_SIZE])
+{
+ struct sha3_256_ctx ctx;
+
+ sha3_256_init(&ctx);
+ sha3_256_update(&ctx, in, in_len);
+ sha3_256_final(&ctx, out);
+}
+EXPORT_SYMBOL(sha3_256);
+
+/**
+ * sha3_384() - Convenience wrapper to digest a simple buffer as SHA3-384
+ * @in: The data to be digested
+ * @in_len: The amount of data to be digested in bytes
+ * @out: The buffer into which the digest will be stored (size not checked)
+ *
+ * Convenience wrapper to initialise a SHA3 context for SHA-384, add the input
+ * data to it, finalise it, extract 48 bytes of digest and clear the context.
+ *
+ * Context: May use the FPU/Vector unit registers.
+ */
+void sha3_384(const u8 *in, size_t in_len, u8 out[SHA3_384_DIGEST_SIZE])
+{
+ struct sha3_384_ctx ctx;
+
+ sha3_384_init(&ctx);
+ sha3_384_update(&ctx, in, in_len);
+ sha3_384_final(&ctx, out);
+}
+EXPORT_SYMBOL(sha3_384);
+
+/**
+ * sha3_512() - Convenience wrapper to digest a simple buffer as SHA3-512
+ * @in: The data to be digested in bytes
+ * @in_len: The amount of data to be digested in bytes
+ * @out: The buffer into which the digest will be stored (size not checked)
+ *
+ * Convenience wrapper to initialise a SHA3 context for SHA-512, add the input
+ * data to it, finalise it, extract 64 bytes of digest and clear the context.
+ *
+ * Context: May use the FPU/Vector unit registers.
+ */
+void sha3_512(const u8 *in, size_t in_len, u8 out[SHA3_512_DIGEST_SIZE])
+{
+ struct sha3_512_ctx ctx;
+
+ sha3_512_init(&ctx);
+ sha3_512_update(&ctx, in, in_len);
+ sha3_512_final(&ctx, out);
+}
+EXPORT_SYMBOL(sha3_512);
+
+/**
+ * shake128() - Convenience wrapper to apply SHAKE128 to a simple buffer
+ * @in: The input data to be used
+ * @in_len: The amount of input data in bytes
+ * @out: The buffer in which to store the output
+ * @out_len: The amount of output to store in bytes (variable length)
+ *
+ * Convenience wrapper to initialise a SHA3 context for SHAKE128, add the input
+ * data to it, finalise it, extract the requested amount of output and clear
+ * the context.
+ *
+ * Context: May use the FPU/Vector unit registers.
+ */
+void shake128(const u8 *in, size_t in_len, u8 *out, size_t out_len)
+{
+ struct shake128_ctx ctx;
+
+ shake128_init(&ctx);
+ shake128_update(&ctx, in, in_len);
+ shake128_squeeze(&ctx, out, out_len);
+ shake128_clear(&ctx);
+}
+EXPORT_SYMBOL(shake128);
+
+/**
+ * shake256() - Convenience wrapper to apply SHAKE256 to a simple buffer
+ * @in: The input data to be used
+ * @in_len: The amount of input data in bytes
+ * @out: The buffer in which to store the output
+ * @out_len: The amount of output to store in bytes (variable length)
+ *
+ * Convenience wrapper to initialise a SHA3 context for SHAKE128, add the input
+ * data to it, finalise it, extract the requested amount of output and clear
+ * the context.
+ *
+ * Context: May use the FPU/Vector unit registers.
+ */
+void shake256(const u8 *in, size_t in_len, u8 *out, size_t out_len)
+{
+ struct shake256_ctx ctx;
+
+ shake256_init(&ctx);
+ shake256_update(&ctx, in, in_len);
+ shake256_squeeze(&ctx, out, out_len);
+ shake256_clear(&ctx);
+}
+EXPORT_SYMBOL(shake256);
+
+/*
+ * Do a quick test using SHAKE256 and a 200 byte digest.
+ */
+static const u8 sha3_sample[] __initconst =
+ "The quick red fox jumped over the lazy brown dog!\n"
+ "The quick red fox jumped over the lazy brown dog!\n"
+ "The quick red fox jumped over the lazy brown dog!\n"
+ "The quick red fox jumped over the lazy brown dog!\n";
+
+static const u8 sha3_sample_shake256_200[] __initconst = {
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* Write-before guard */
+ 0xab, 0x06, 0xd4, 0xf9, 0x8b, 0xfd, 0xb2, 0xc4, 0xfe, 0xf1, 0xcc, 0xe2,
+ 0x40, 0x45, 0xdd, 0x15, 0xcb, 0xdd, 0x02, 0x8d, 0xb7, 0x9f, 0x1e, 0x67,
+ 0xd6, 0x7f, 0x98, 0x5e, 0x1b, 0x19, 0xf8, 0x01, 0x43, 0x82, 0xcb, 0xd8,
+ 0x5d, 0x21, 0x64, 0xa8, 0x80, 0xc9, 0x22, 0xe5, 0x07, 0xaf, 0xe2, 0x5d,
+ 0xcd, 0xc6, 0x23, 0x36, 0x2b, 0xc7, 0xc7, 0x7d, 0x09, 0x9d, 0x68, 0x05,
+ 0xe4, 0x62, 0x63, 0x1b, 0x67, 0xbc, 0xf8, 0x95, 0x07, 0xd2, 0xe4, 0xd0,
+ 0xba, 0xa2, 0x67, 0xf5, 0xe3, 0x15, 0xbc, 0x85, 0xa1, 0x50, 0xd6, 0x6f,
+ 0x6f, 0xd4, 0x54, 0x4c, 0x3f, 0x4f, 0xe5, 0x1f, 0xb7, 0x00, 0x27, 0xfc,
+ 0x15, 0x33, 0xc2, 0xf9, 0xb3, 0x4b, 0x9e, 0x81, 0xe5, 0x96, 0xbe, 0x05,
+ 0x6c, 0xac, 0xf9, 0x9f, 0x65, 0x36, 0xbb, 0x11, 0x47, 0x6d, 0xf6, 0x8f,
+ 0x9f, 0xa2, 0x77, 0x37, 0x3b, 0x18, 0x77, 0xcf, 0x65, 0xc5, 0xa1, 0x7e,
+ 0x2c, 0x0e, 0x71, 0xf0, 0x4d, 0x18, 0x67, 0xb9, 0xc4, 0x8c, 0x64, 0x3b,
+ 0x4b, 0x45, 0xea, 0x16, 0xb2, 0x4a, 0xc5, 0xf5, 0x85, 0xdc, 0xd2, 0xd9,
+ 0x13, 0x77, 0xb3, 0x19, 0xd9, 0x8c, 0x9f, 0x28, 0xe7, 0x64, 0x91, 0x0f,
+ 0x6f, 0x32, 0xbf, 0xa8, 0xa8, 0xa3, 0xff, 0x99, 0x0e, 0x0b, 0x62, 0x50,
+ 0xf8, 0x3a, 0xc2, 0xf5, 0x98, 0x21, 0xeb, 0x9d, 0xe8, 0x45, 0xf4, 0x46,
+ 0x1e, 0x8b, 0xbd, 0x10, 0x59, 0x2c, 0x87, 0xe2,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* Write-after guard */
+};
+
+static int __init sha3_mod_init(void)
+{
+#define out_len 200
+ u8 out[8 + out_len + 8] = {};
+
+#ifdef sha3_mod_init_arch
+ sha3_mod_init_arch();
+#endif
+
+ BUILD_BUG_ON(sizeof(out) != sizeof(sha3_sample_shake256_200));
+
+ shake256(sha3_sample, sizeof(sha3_sample) - 1, out + 8, out_len);
+
+ if (memcmp(out, sha3_sample_shake256_200,
+ sizeof(sha3_sample_shake256_200)) != 0) {
+ pr_err("SHAKE256(200) failed\n");
+ for (size_t i = 0; i < out_len;) {
+ size_t part = min(out_len - i, 32);
+
+ pr_err("%*phN\n", (int)part, out + i);
+ i += part;
+ }
+ return -EBADMSG;
+ }
+ return 0;
+}
+subsys_initcall(sha3_mod_init);
+
+static void __exit sha3_mod_exit(void)
+{
+}
+module_exit(sha3_mod_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("SHA-3 Secure Hash Algorithm");
^ permalink raw reply related
* [PATCH v6 04/17] lib/crypto: Move the SHA3 Iota transform into the single round function
From: David Howells @ 2025-10-17 14:42 UTC (permalink / raw)
To: Eric Biggers
Cc: David Howells, Jason A . Donenfeld, Ard Biesheuvel, Herbert Xu,
Stephan Mueller, Lukas Wunner, Ignat Korchagin, Luis Chamberlain,
Petr Pavlu, Daniel Gomez, Sami Tolvanen, linux-crypto, keyrings,
linux-modules, linux-kernel
In-Reply-To: <20251017144311.817771-1-dhowells@redhat.com>
In crypto/sha3_generic.c, the keccakf() function calls keccakf_round() to
do most of the transforms, but not the Iota transform - presumably because
that is dependent on round number, whereas the Theta, Rho, Pi and Chi
transforms are not.
Note that the keccakf_round() function needs to be explicitly non-inlined
on certain architectures as gcc's produced output will (or used to) use
over 1KiB of stack space if inlined.
Now, this code was copied more or less verbatim into lib/crypto/sha3.c, so
that has the same aesthetic issue. Fix this there by passing the round
number into sha3_keccakf_one_round_generic() and doing the Iota transform
there.
crypto/sha3_generic.c is left untouched as that will be converted to use
lib/crypto/sha3.c at some point.
Suggested-by: Eric Biggers <ebiggers@kernel.org>
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Eric Biggers <ebiggers@kernel.org>
cc: Jason A. Donenfeld <Jason@zx2c4.com>
cc: Ard Biesheuvel <ardb@kernel.org>
cc: Herbert Xu <herbert@gondor.apana.org.au>
cc: Stephan Mueller <smueller@chronox.de>
cc: linux-crypto@vger.kernel.org
---
lib/crypto/sha3.c | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/lib/crypto/sha3.c b/lib/crypto/sha3.c
index 5f847a5eecdc..2c292b0b3db3 100644
--- a/lib/crypto/sha3.c
+++ b/lib/crypto/sha3.c
@@ -48,7 +48,8 @@ static const u64 sha3_keccakf_rndc[24] = {
/*
* Perform a single round of Keccak mixing.
*/
-static SHA3_INLINE void sha3_keccakf_one_round_generic(struct sha3_state *state)
+static SHA3_INLINE void sha3_keccakf_one_round_generic(struct sha3_state *state,
+ int round)
{
u64 *st = state->st;
u64 t[5], tt, bc[5];
@@ -150,15 +151,15 @@ static SHA3_INLINE void sha3_keccakf_one_round_generic(struct sha3_state *state)
st[22] ^= bc[ 2];
st[23] ^= bc[ 3];
st[24] ^= bc[ 4];
+
+ /* Iota */
+ state->st[0] ^= sha3_keccakf_rndc[round];
}
static void sha3_keccakf_rounds_generic(struct sha3_state *state)
{
- for (int round = 0; round < SHA3_KECCAK_ROUNDS; round++) {
- sha3_keccakf_one_round_generic(state);
- /* Iota */
- state->st[0] ^= sha3_keccakf_rndc[round];
- }
+ for (int round = 0; round < SHA3_KECCAK_ROUNDS; round++)
+ sha3_keccakf_one_round_generic(state, round);
}
/*
^ permalink raw reply related
* [PATCH v6 05/17] lib/crypto: Add SHA3 kunit tests
From: David Howells @ 2025-10-17 14:42 UTC (permalink / raw)
To: Eric Biggers
Cc: David Howells, Jason A . Donenfeld, Ard Biesheuvel, Herbert Xu,
Stephan Mueller, Lukas Wunner, Ignat Korchagin, Luis Chamberlain,
Petr Pavlu, Daniel Gomez, Sami Tolvanen, linux-crypto, keyrings,
linux-modules, linux-kernel
In-Reply-To: <20251017144311.817771-1-dhowells@redhat.com>
Add a SHA3 kunit test, providing the following:
(*) A simple test of each of SHA3-224, SHA3-256, SHA3-384, SHA-512,
SHAKE128 and SHAKE256.
(*) NIST 0- and 1600-bit test vectors for SHAKE128 and SHAKE256.
(*) Output tiling (multiple squeezing) tests for SHAKE256.
(*) Standard hash template test for SHA3-256.
(*) Standard benchmark test for SHA3-256.
gen-hash-testvecs.py is also modified in a number of ways:
(1) To be able to generate SHAKE* testvecs because Python's hashlib
requires the output digest size supplying for those two algorithms as
they produce arbitrary length digests.
(2) To change '-' in the algo name into '_' when generating symbols from
it.
(3) To not generate HMAC for SHA3 or SHAKE as support for that isn't
currently implemented.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Eric Biggers <ebiggers@kernel.org>
cc: Jason A. Donenfeld <Jason@zx2c4.com>
cc: Ard Biesheuvel <ardb@kernel.org>
cc: Herbert Xu <herbert@gondor.apana.org.au>
cc: Stephan Mueller <smueller@chronox.de>
cc: linux-crypto@vger.kernel.org
Changes
=======
v5)
- Fix gen-hash-testvecs.py to correctly handle algo names that contain a
dash.
- Fix gen-hash-testvecs.py to not generate HMAC for SHA3-* or SHAKE* as
these don't currently have HMAC variants.
- Fix algo names to be correct.
- Fix kunit module description as it now tests all SHA3 variants.
---
lib/crypto/tests/Kconfig | 11 +
lib/crypto/tests/Makefile | 1 +
lib/crypto/tests/sha3_kunit.c | 342 ++++++++++++++++++++++++++++
lib/crypto/tests/sha3_testvecs.h | 231 +++++++++++++++++++
scripts/crypto/gen-hash-testvecs.py | 10 +-
5 files changed, 593 insertions(+), 2 deletions(-)
create mode 100644 lib/crypto/tests/sha3_kunit.c
create mode 100644 lib/crypto/tests/sha3_testvecs.h
diff --git a/lib/crypto/tests/Kconfig b/lib/crypto/tests/Kconfig
index 578af717e13a..67538968edab 100644
--- a/lib/crypto/tests/Kconfig
+++ b/lib/crypto/tests/Kconfig
@@ -72,6 +72,17 @@ config CRYPTO_LIB_SHA512_KUNIT_TEST
KUnit tests for the SHA-384 and SHA-512 cryptographic hash functions
and their corresponding HMACs.
+config CRYPTO_LIB_SHA3_KUNIT_TEST
+ tristate "KUnit tests for SHA-3" if !KUNIT_ALL_TESTS
+ depends on KUNIT
+ default KUNIT_ALL_TESTS || CRYPTO_SELFTESTS
+ select CRYPTO_LIB_BENCHMARK_VISIBLE
+ select CRYPTO_LIB_SHA3
+ help
+ KUnit tests for the SHA3 cryptographic hash and XOF functions,
+ including SHA3-224, SHA3-256, SHA3-384, SHA3-512, SHAKE128 and
+ SHAKE256.
+
config CRYPTO_LIB_BENCHMARK_VISIBLE
bool
diff --git a/lib/crypto/tests/Makefile b/lib/crypto/tests/Makefile
index a71fad19922b..9c61b29ac4ca 100644
--- a/lib/crypto/tests/Makefile
+++ b/lib/crypto/tests/Makefile
@@ -7,3 +7,4 @@ obj-$(CONFIG_CRYPTO_LIB_POLY1305_KUNIT_TEST) += poly1305_kunit.o
obj-$(CONFIG_CRYPTO_LIB_SHA1_KUNIT_TEST) += sha1_kunit.o
obj-$(CONFIG_CRYPTO_LIB_SHA256_KUNIT_TEST) += sha224_kunit.o sha256_kunit.o
obj-$(CONFIG_CRYPTO_LIB_SHA512_KUNIT_TEST) += sha384_kunit.o sha512_kunit.o
+obj-$(CONFIG_CRYPTO_LIB_SHA3_KUNIT_TEST) += sha3_kunit.o
diff --git a/lib/crypto/tests/sha3_kunit.c b/lib/crypto/tests/sha3_kunit.c
new file mode 100644
index 000000000000..032fa3950afe
--- /dev/null
+++ b/lib/crypto/tests/sha3_kunit.c
@@ -0,0 +1,342 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2025 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells (dhowells@redhat.com)
+ */
+#include <crypto/sha3.h>
+#include "sha3_testvecs.h"
+
+#define HASH sha3_256
+#define HASH_CTX sha3_256_ctx
+#define HASH_SIZE SHA3_256_DIGEST_SIZE
+#define HASH_INIT sha3_256_init
+#define HASH_UPDATE sha3_256_update
+#define HASH_FINAL sha3_256_final
+#include "hash-test-template.h"
+
+/*
+ * Sample message and the output generated for various algorithms by passing it
+ * into "openssl sha3-224" etc..
+ */
+static const u8 test_sha3_sample[] =
+ "The quick red fox jumped over the lazy brown dog!\n"
+ "The quick red fox jumped over the lazy brown dog!\n"
+ "The quick red fox jumped over the lazy brown dog!\n"
+ "The quick red fox jumped over the lazy brown dog!\n";
+
+static const u8 test_sha3_224[8 + SHA3_224_DIGEST_SIZE + 8] = {
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* Write-before guard */
+ 0xd6, 0xe8, 0xd8, 0x80, 0xfa, 0x42, 0x80, 0x70,
+ 0x7e, 0x7f, 0xd7, 0xd2, 0xd7, 0x7a, 0x35, 0x65,
+ 0xf0, 0x0b, 0x4f, 0x9f, 0x2a, 0x33, 0xca, 0x0a,
+ 0xef, 0xa6, 0x4c, 0xb8,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* Write-after guard */
+};
+
+static const u8 test_sha3_256[8 + SHA3_256_DIGEST_SIZE + 8] = {
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* Write-before guard */
+ 0xdb, 0x3b, 0xb0, 0xb8, 0x8d, 0x15, 0x78, 0xe5,
+ 0x78, 0x76, 0x8e, 0x39, 0x7e, 0x89, 0x86, 0xb9,
+ 0x14, 0x3a, 0x1e, 0xe7, 0x96, 0x7c, 0xf3, 0x25,
+ 0x70, 0xbd, 0xc3, 0xa9, 0xae, 0x63, 0x71, 0x1d,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* Write-after guard */
+};
+
+static const u8 test_sha3_384[8 + SHA3_384_DIGEST_SIZE + 8] = {
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* Write-before guard */
+ 0x2d, 0x4b, 0x29, 0x85, 0x19, 0x94, 0xaa, 0x31,
+ 0x9b, 0x04, 0x9d, 0x6e, 0x79, 0x66, 0xc7, 0x56,
+ 0x8a, 0x2e, 0x99, 0x84, 0x06, 0xcf, 0x10, 0x2d,
+ 0xec, 0xf0, 0x03, 0x04, 0x1f, 0xd5, 0x99, 0x63,
+ 0x2f, 0xc3, 0x2b, 0x0d, 0xd9, 0x45, 0xf7, 0xbb,
+ 0x0a, 0xc3, 0x46, 0xab, 0xfe, 0x4d, 0x94, 0xc2,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* Write-after guard */
+};
+
+static const u8 test_sha3_512[8 + SHA3_512_DIGEST_SIZE + 8] = {
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* Write-before guard */
+ 0xdd, 0x71, 0x3b, 0x44, 0xb6, 0x6c, 0xd7, 0x78,
+ 0xe7, 0x93, 0xa1, 0x4c, 0xd7, 0x24, 0x16, 0xf1,
+ 0xfd, 0xa2, 0x82, 0x4e, 0xed, 0x59, 0xe9, 0x83,
+ 0x15, 0x38, 0x89, 0x7d, 0x39, 0x17, 0x0c, 0xb2,
+ 0xcf, 0x12, 0x80, 0x78, 0xa1, 0x78, 0x41, 0xeb,
+ 0xed, 0x21, 0x4c, 0xa4, 0x4a, 0x5f, 0x30, 0x1a,
+ 0x70, 0x98, 0x4f, 0x14, 0xa2, 0xd1, 0x64, 0x1b,
+ 0xc2, 0x0a, 0xff, 0x3b, 0xe8, 0x26, 0x41, 0x8f,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* Write-after guard */
+};
+
+static const u8 test_shake128[8 + SHAKE128_DEFAULT_SIZE + 8] = {
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* Write-before guard */
+ 0x41, 0xd6, 0xb8, 0x9c, 0xf8, 0xe8, 0x54, 0xf2,
+ 0x5c, 0xde, 0x51, 0x12, 0xaf, 0x9e, 0x0d, 0x91,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* Write-after guard */
+};
+
+static const u8 test_shake256[8 + SHAKE256_DEFAULT_SIZE + 8] = {
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* Write-before guard */
+ 0xab, 0x06, 0xd4, 0xf9, 0x8b, 0xfd, 0xb2, 0xc4,
+ 0xfe, 0xf1, 0xcc, 0xe2, 0x40, 0x45, 0xdd, 0x15,
+ 0xcb, 0xdd, 0x02, 0x8d, 0xb7, 0x9f, 0x1e, 0x67,
+ 0xd6, 0x7f, 0x98, 0x5e, 0x1b, 0x19, 0xf8, 0x01,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* Write-after guard */
+};
+
+static void test_sha3_224_basic(struct kunit *test)
+{
+ u8 out[8 + SHA3_224_DIGEST_SIZE + 8];
+
+ BUILD_BUG_ON(sizeof(out) != sizeof(test_sha3_224));
+
+ memset(out, 0, sizeof(out));
+ sha3_224(test_sha3_sample, sizeof(test_sha3_sample) - 1, out + 8);
+
+ KUNIT_ASSERT_MEMEQ_MSG(test, out, test_sha3_224, sizeof(test_sha3_224),
+ "SHA3-224 gives wrong output");
+}
+
+static void test_sha3_256_basic(struct kunit *test)
+{
+ u8 out[8 + SHA3_256_DIGEST_SIZE + 8];
+
+ BUILD_BUG_ON(sizeof(out) != sizeof(test_sha3_256));
+
+ memset(out, 0, sizeof(out));
+ sha3_256(test_sha3_sample, sizeof(test_sha3_sample) - 1, out + 8);
+
+ KUNIT_ASSERT_MEMEQ_MSG(test, out, test_sha3_256, sizeof(test_sha3_256),
+ "SHA3-256 gives wrong output");
+}
+
+static void test_sha3_384_basic(struct kunit *test)
+{
+ u8 out[8 + SHA3_384_DIGEST_SIZE + 8];
+
+ BUILD_BUG_ON(sizeof(out) != sizeof(test_sha3_384));
+
+ memset(out, 0, sizeof(out));
+ sha3_384(test_sha3_sample, sizeof(test_sha3_sample) - 1, out + 8);
+
+ KUNIT_ASSERT_MEMEQ_MSG(test, out, test_sha3_384, sizeof(test_sha3_384),
+ "SHA3-384 gives wrong output");
+}
+
+static void test_sha3_512_basic(struct kunit *test)
+{
+ u8 out[8 + SHA3_512_DIGEST_SIZE + 8];
+
+ BUILD_BUG_ON(sizeof(out) != sizeof(test_sha3_512));
+
+ memset(out, 0, sizeof(out));
+ sha3_512(test_sha3_sample, sizeof(test_sha3_sample) - 1, out + 8);
+
+ KUNIT_ASSERT_MEMEQ_MSG(test, out, test_sha3_512, sizeof(test_sha3_512),
+ "SHA3-512 gives wrong output");
+}
+
+static void test_shake128_basic(struct kunit *test)
+{
+ u8 out[8 + SHAKE128_DEFAULT_SIZE + 8];
+
+ BUILD_BUG_ON(sizeof(out) != sizeof(test_shake128));
+
+ memset(out, 0, sizeof(out));
+ shake128(test_sha3_sample, sizeof(test_sha3_sample) - 1,
+ out + 8, sizeof(out) - 16);
+
+ KUNIT_ASSERT_MEMEQ_MSG(test, out, test_shake128, sizeof(test_shake128),
+ "SHAKE128 gives wrong output");
+}
+
+static void test_shake256_basic(struct kunit *test)
+{
+ u8 out[8 + SHAKE256_DEFAULT_SIZE + 8];
+
+ BUILD_BUG_ON(sizeof(out) != sizeof(test_shake256));
+
+ memset(out, 0, sizeof(out));
+ shake256(test_sha3_sample, sizeof(test_sha3_sample) - 1,
+ out + 8, sizeof(out) - 16);
+
+ KUNIT_ASSERT_MEMEQ_MSG(test, out, test_shake256, sizeof(test_shake256),
+ "SHAKE256 gives wrong output");
+}
+
+/*
+ * Usable NIST tests.
+ *
+ * From: https://csrc.nist.gov/projects/cryptographic-standards-and-guidelines/example-values
+ */
+static const u8 test_nist_1600_sample[] = {
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3,
+ 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3, 0xa3
+};
+
+static const u8 test_shake128_nist_0[] = {
+ 0x7f, 0x9c, 0x2b, 0xa4, 0xe8, 0x8f, 0x82, 0x7d,
+ 0x61, 0x60, 0x45, 0x50, 0x76, 0x05, 0x85, 0x3e
+};
+
+static const u8 test_shake128_nist_1600[] = {
+ 0x13, 0x1a, 0xb8, 0xd2, 0xb5, 0x94, 0x94, 0x6b,
+ 0x9c, 0x81, 0x33, 0x3f, 0x9b, 0xb6, 0xe0, 0xce,
+};
+
+static const u8 test_shake256_nist_0[] = {
+ 0x46, 0xb9, 0xdd, 0x2b, 0x0b, 0xa8, 0x8d, 0x13,
+ 0x23, 0x3b, 0x3f, 0xeb, 0x74, 0x3e, 0xeb, 0x24,
+ 0x3f, 0xcd, 0x52, 0xea, 0x62, 0xb8, 0x1b, 0x82,
+ 0xb5, 0x0c, 0x27, 0x64, 0x6e, 0xd5, 0x76, 0x2f
+};
+
+static const u8 test_shake256_nist_1600[] = {
+ 0xcd, 0x8a, 0x92, 0x0e, 0xd1, 0x41, 0xaa, 0x04,
+ 0x07, 0xa2, 0x2d, 0x59, 0x28, 0x86, 0x52, 0xe9,
+ 0xd9, 0xf1, 0xa7, 0xee, 0x0c, 0x1e, 0x7c, 0x1c,
+ 0xa6, 0x99, 0x42, 0x4d, 0xa8, 0x4a, 0x90, 0x4d,
+};
+
+static void test_shake128_nist(struct kunit *test)
+{
+ u8 out[SHAKE128_DEFAULT_SIZE];
+
+ shake128("", 0, out, sizeof(out));
+ KUNIT_ASSERT_MEMEQ_MSG(test, out, test_shake128_nist_0, sizeof(out),
+ "SHAKE128 gives wrong output for NIST.0");
+
+ shake128(test_nist_1600_sample, sizeof(test_nist_1600_sample),
+ out, sizeof(out));
+ KUNIT_ASSERT_MEMEQ_MSG(test, out, test_shake128_nist_1600, sizeof(out),
+ "SHAKE128 gives wrong output for NIST.1600");
+}
+
+static void test_shake256_nist(struct kunit *test)
+{
+ u8 out[SHAKE256_DEFAULT_SIZE];
+
+ shake256("", 0, out, sizeof(out));
+ KUNIT_ASSERT_MEMEQ_MSG(test, out, test_shake256_nist_0, sizeof(out),
+ "SHAKE256 gives wrong output for NIST.0");
+
+ shake256(test_nist_1600_sample, sizeof(test_nist_1600_sample),
+ out, sizeof(out));
+ KUNIT_ASSERT_MEMEQ_MSG(test, out, test_shake256_nist_1600, sizeof(out),
+ "SHAKE256 gives wrong output for NIST.1600");
+}
+
+/*
+ * Output tiling test of SHAKE256; equal output tiles barring the last. A
+ * series of squeezings of the same context should, if laid end-to-end, match a
+ * single squeezing of the combined size.
+ */
+static void test_shake256_tiling(struct kunit *test)
+{
+ struct shake256_ctx ctx;
+ u8 out[8 + SHA3_512_DIGEST_SIZE + 8];
+
+ for (int tile_size = 1; tile_size < SHAKE256_DEFAULT_SIZE; tile_size++) {
+ int left = SHAKE256_DEFAULT_SIZE;
+ u8 *p = out + 8;
+
+ memset(out, 0, sizeof(out));
+ shake256_init(&ctx);
+ shake256_update(&ctx, test_sha3_sample, sizeof(test_sha3_sample) - 1);
+ while (left > 0) {
+ int part = umin(tile_size, left);
+
+ shake256_squeeze(&ctx, p, part);
+ p += part;
+ left -= part;
+ }
+
+ KUNIT_ASSERT_MEMEQ_MSG(test, out, test_shake256, sizeof(test_shake256),
+ "SHAKE tile %u gives wrong output", tile_size);
+ }
+}
+
+/*
+ * Output tiling test of SHAKE256; output tiles getting gradually smaller and
+ * then cycling round to medium sized ones. A series of squeezings of the same
+ * context should, if laid end-to-end, match a single squeezing of the combined
+ * size.
+ */
+static void test_shake256_tiling2(struct kunit *test)
+{
+ struct shake256_ctx ctx;
+ u8 out[8 + SHA3_512_DIGEST_SIZE + 8];
+
+ for (int first_tile_size = 3;
+ first_tile_size < SHAKE256_DEFAULT_SIZE;
+ first_tile_size++) {
+ int tile_size = first_tile_size;
+ int left = SHAKE256_DEFAULT_SIZE;
+ u8 *p = out + 8;
+
+ memset(out, 0, sizeof(out));
+ shake256_init(&ctx);
+ shake256_update(&ctx, test_sha3_sample, sizeof(test_sha3_sample) - 1);
+ while (left > 0) {
+ int part = umin(tile_size, left);
+
+ shake256_squeeze(&ctx, p, part);
+ p += part;
+ left -= part;
+ tile_size--;
+ if (tile_size < 1)
+ tile_size = 5;
+ }
+
+ KUNIT_ASSERT_MEMEQ_MSG(test, out, test_shake256, sizeof(test_shake256),
+ "SHAKE tile %u gives wrong output", tile_size);
+ }
+}
+
+static struct kunit_case hash_test_cases[] = {
+ KUNIT_CASE(test_sha3_224_basic),
+ KUNIT_CASE(test_sha3_256_basic),
+ KUNIT_CASE(test_sha3_384_basic),
+ KUNIT_CASE(test_sha3_512_basic),
+ KUNIT_CASE(test_shake128_basic),
+ KUNIT_CASE(test_shake256_basic),
+ KUNIT_CASE(test_shake128_nist),
+ KUNIT_CASE(test_shake256_nist),
+ KUNIT_CASE(test_shake256_tiling),
+ KUNIT_CASE(test_shake256_tiling2),
+ HASH_KUNIT_CASES,
+ KUNIT_CASE(benchmark_hash),
+ {},
+};
+
+static struct kunit_suite hash_test_suite = {
+ .name = "sha3",
+ .test_cases = hash_test_cases,
+ .suite_init = hash_suite_init,
+ .suite_exit = hash_suite_exit,
+};
+kunit_test_suite(hash_test_suite);
+
+MODULE_DESCRIPTION("KUnit tests and benchmark for SHA3");
+MODULE_LICENSE("GPL");
diff --git a/lib/crypto/tests/sha3_testvecs.h b/lib/crypto/tests/sha3_testvecs.h
new file mode 100644
index 000000000000..9c4c403cc6e0
--- /dev/null
+++ b/lib/crypto/tests/sha3_testvecs.h
@@ -0,0 +1,231 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/* This file was generated by: ./scripts/crypto/gen-hash-testvecs.py sha3-256 */
+
+static const struct {
+ size_t data_len;
+ u8 digest[SHA3_256_DIGEST_SIZE];
+} hash_testvecs[] = {
+ {
+ .data_len = 0,
+ .digest = {
+ 0xa7, 0xff, 0xc6, 0xf8, 0xbf, 0x1e, 0xd7, 0x66,
+ 0x51, 0xc1, 0x47, 0x56, 0xa0, 0x61, 0xd6, 0x62,
+ 0xf5, 0x80, 0xff, 0x4d, 0xe4, 0x3b, 0x49, 0xfa,
+ 0x82, 0xd8, 0x0a, 0x4b, 0x80, 0xf8, 0x43, 0x4a,
+ },
+ },
+ {
+ .data_len = 1,
+ .digest = {
+ 0x11, 0x03, 0xe7, 0x84, 0x51, 0x50, 0x86, 0x35,
+ 0x71, 0x8a, 0x70, 0xe3, 0xc4, 0x26, 0x7b, 0x21,
+ 0x02, 0x13, 0xa0, 0x81, 0xe8, 0xe6, 0x14, 0x25,
+ 0x07, 0x34, 0xe5, 0xc5, 0x40, 0x06, 0xf2, 0x8b,
+ },
+ },
+ {
+ .data_len = 2,
+ .digest = {
+ 0x2f, 0x6f, 0x6d, 0x47, 0x48, 0x52, 0x11, 0xb9,
+ 0xe4, 0x3d, 0xc8, 0x71, 0xcf, 0xb2, 0xee, 0xae,
+ 0x5b, 0xf4, 0x12, 0x84, 0x5b, 0x1c, 0xec, 0x6c,
+ 0xc1, 0x66, 0x88, 0xaa, 0xc3, 0x40, 0xbd, 0x7e,
+ },
+ },
+ {
+ .data_len = 3,
+ .digest = {
+ 0xec, 0x02, 0xe8, 0x81, 0x4f, 0x84, 0x41, 0x69,
+ 0x06, 0xd8, 0xdc, 0x1d, 0x01, 0x78, 0xd7, 0xcb,
+ 0x39, 0xdf, 0xd3, 0x12, 0x1c, 0x99, 0xfd, 0xf3,
+ 0x5c, 0x83, 0xc9, 0xc2, 0x7a, 0x7b, 0x6a, 0x05,
+ },
+ },
+ {
+ .data_len = 16,
+ .digest = {
+ 0xff, 0x6f, 0xc3, 0x41, 0xc3, 0x5f, 0x34, 0x6d,
+ 0xa7, 0xdf, 0x3e, 0xc2, 0x8b, 0x29, 0xb6, 0xf1,
+ 0xf8, 0x67, 0xfd, 0xcd, 0xb1, 0x9f, 0x38, 0x08,
+ 0x1d, 0x8d, 0xd9, 0xc2, 0x43, 0x66, 0x18, 0x6c,
+ },
+ },
+ {
+ .data_len = 32,
+ .digest = {
+ 0xe4, 0xb1, 0x06, 0x17, 0xf8, 0x8b, 0x91, 0x95,
+ 0xe7, 0x57, 0x66, 0xac, 0x08, 0xb2, 0x03, 0x3e,
+ 0xf7, 0x84, 0x1f, 0xe3, 0x25, 0xa3, 0x11, 0xd2,
+ 0x11, 0xa4, 0x78, 0x74, 0x2a, 0x43, 0x20, 0xa5,
+ },
+ },
+ {
+ .data_len = 48,
+ .digest = {
+ 0xeb, 0x57, 0x5f, 0x20, 0xa3, 0x6b, 0xc7, 0xb4,
+ 0x66, 0x2a, 0xa0, 0x30, 0x3b, 0x52, 0x00, 0xc9,
+ 0xce, 0x6a, 0xd8, 0x1e, 0xbe, 0xed, 0xa1, 0xd1,
+ 0xbe, 0x63, 0xc7, 0xe1, 0xe2, 0x66, 0x67, 0x0c,
+ },
+ },
+ {
+ .data_len = 49,
+ .digest = {
+ 0xf0, 0x67, 0xad, 0x66, 0xbe, 0xec, 0x5a, 0xfd,
+ 0x29, 0xd2, 0x4f, 0x1d, 0xb2, 0x24, 0xb8, 0x90,
+ 0x05, 0x28, 0x0e, 0x66, 0x67, 0x74, 0x2d, 0xee,
+ 0x66, 0x25, 0x11, 0xd1, 0x76, 0xa2, 0xfc, 0x3a,
+ },
+ },
+ {
+ .data_len = 63,
+ .digest = {
+ 0x57, 0x56, 0x21, 0xb3, 0x2d, 0x2d, 0xe1, 0x9d,
+ 0xbf, 0x2c, 0x82, 0xa8, 0xad, 0x7e, 0x6c, 0x46,
+ 0xfb, 0x30, 0xeb, 0xce, 0xcf, 0xed, 0x2d, 0x65,
+ 0xe7, 0xe4, 0x96, 0x69, 0xe0, 0x48, 0xd2, 0xb6,
+ },
+ },
+ {
+ .data_len = 64,
+ .digest = {
+ 0x7b, 0xba, 0x67, 0x15, 0xe5, 0x21, 0xc4, 0x69,
+ 0xd3, 0xef, 0x5c, 0x97, 0x9f, 0x5b, 0xba, 0x9c,
+ 0xfa, 0x55, 0x64, 0xec, 0xb5, 0x37, 0x53, 0x1b,
+ 0x3f, 0x4c, 0x0a, 0xed, 0x51, 0x98, 0x2b, 0x52,
+ },
+ },
+ {
+ .data_len = 65,
+ .digest = {
+ 0x44, 0xb6, 0x6b, 0x83, 0x09, 0x83, 0x55, 0x83,
+ 0xde, 0x1f, 0xcc, 0x33, 0xef, 0xdc, 0x05, 0xbb,
+ 0x3b, 0x63, 0x76, 0x45, 0xe4, 0x8e, 0x14, 0x7a,
+ 0x2d, 0xae, 0x90, 0xce, 0x68, 0xc3, 0xa4, 0xf2,
+ },
+ },
+ {
+ .data_len = 127,
+ .digest = {
+ 0x50, 0x3e, 0x99, 0x4e, 0x28, 0x2b, 0xc9, 0xf4,
+ 0xf5, 0xeb, 0x2b, 0x16, 0x04, 0x2d, 0xf5, 0xbe,
+ 0xc0, 0x91, 0x41, 0x2a, 0x8e, 0x69, 0x5e, 0x39,
+ 0x53, 0x2c, 0xc1, 0x18, 0xa5, 0xeb, 0xd8, 0xda,
+ },
+ },
+ {
+ .data_len = 128,
+ .digest = {
+ 0x90, 0x0b, 0xa6, 0x92, 0x84, 0x30, 0xaf, 0xee,
+ 0x38, 0x59, 0x83, 0x83, 0xe9, 0xfe, 0xab, 0x86,
+ 0x79, 0x1b, 0xcd, 0xe7, 0x0a, 0x0f, 0x58, 0x53,
+ 0x36, 0xab, 0x12, 0xe1, 0x5c, 0x97, 0xc1, 0xfb,
+ },
+ },
+ {
+ .data_len = 129,
+ .digest = {
+ 0x2b, 0x52, 0x1e, 0x54, 0xbe, 0x38, 0x4c, 0x3e,
+ 0x73, 0x37, 0x18, 0xf5, 0x25, 0x2c, 0xc8, 0xc7,
+ 0xda, 0x7e, 0xb6, 0x47, 0x9d, 0xf4, 0x46, 0xce,
+ 0xfa, 0x80, 0x20, 0x6b, 0xbd, 0xfd, 0x2a, 0xd8,
+ },
+ },
+ {
+ .data_len = 256,
+ .digest = {
+ 0x45, 0xf0, 0xf5, 0x9b, 0xd9, 0x91, 0x26, 0xd5,
+ 0x91, 0x3b, 0xf8, 0x87, 0x8b, 0x34, 0x02, 0x31,
+ 0x64, 0xab, 0xf4, 0x1c, 0x6e, 0x34, 0x72, 0xdf,
+ 0x32, 0x6d, 0xe5, 0xd2, 0x67, 0x5e, 0x86, 0x93,
+ },
+ },
+ {
+ .data_len = 511,
+ .digest = {
+ 0xb3, 0xaf, 0x71, 0x64, 0xfa, 0xd4, 0xf1, 0x07,
+ 0x38, 0xef, 0x04, 0x8e, 0x89, 0xf4, 0x02, 0xd2,
+ 0xa5, 0xaf, 0x3b, 0xf5, 0x67, 0x56, 0xcf, 0xa9,
+ 0x8e, 0x43, 0xf5, 0xb5, 0xe3, 0x91, 0x8e, 0xe7,
+ },
+ },
+ {
+ .data_len = 513,
+ .digest = {
+ 0x51, 0xac, 0x0a, 0x65, 0xb7, 0x96, 0x20, 0xcf,
+ 0x88, 0xf6, 0x97, 0x35, 0x89, 0x0d, 0x31, 0x0f,
+ 0xbe, 0x17, 0xbe, 0x62, 0x03, 0x67, 0xc0, 0xee,
+ 0x4f, 0xc1, 0xe3, 0x7f, 0x6f, 0xab, 0xac, 0xb4,
+ },
+ },
+ {
+ .data_len = 1000,
+ .digest = {
+ 0x7e, 0xea, 0xa8, 0xd7, 0xde, 0x20, 0x1b, 0x58,
+ 0x24, 0xd8, 0x26, 0x40, 0x36, 0x5f, 0x3f, 0xaa,
+ 0xe5, 0x5a, 0xea, 0x98, 0x58, 0xd4, 0xd6, 0xfc,
+ 0x20, 0x4c, 0x5c, 0x4f, 0xaf, 0x56, 0xc7, 0xc3,
+ },
+ },
+ {
+ .data_len = 3333,
+ .digest = {
+ 0x61, 0xb1, 0xb1, 0x3e, 0x0e, 0x7e, 0x90, 0x3d,
+ 0x31, 0x54, 0xbd, 0xc9, 0x0d, 0x53, 0x62, 0xf1,
+ 0xcd, 0x18, 0x80, 0xf9, 0x91, 0x75, 0x41, 0xb3,
+ 0x51, 0x39, 0x57, 0xa7, 0xa8, 0x1e, 0xfb, 0xc9,
+ },
+ },
+ {
+ .data_len = 4096,
+ .digest = {
+ 0xab, 0x29, 0xda, 0x10, 0xc4, 0x11, 0x2d, 0x5c,
+ 0xd1, 0xce, 0x1c, 0x95, 0xfa, 0xc6, 0xc7, 0xb0,
+ 0x1b, 0xd1, 0xdc, 0x6f, 0xa0, 0x9d, 0x1b, 0x23,
+ 0xfb, 0x6e, 0x90, 0x97, 0xd0, 0x75, 0x44, 0x7a,
+ },
+ },
+ {
+ .data_len = 4128,
+ .digest = {
+ 0x02, 0x45, 0x95, 0xf4, 0x19, 0xb5, 0x93, 0x29,
+ 0x90, 0xf2, 0x63, 0x3f, 0x89, 0xe8, 0xa5, 0x31,
+ 0x76, 0xf2, 0x89, 0x79, 0x66, 0xd3, 0x96, 0xdf,
+ 0x33, 0xd1, 0xa6, 0x17, 0x73, 0xb1, 0xd0, 0x45,
+ },
+ },
+ {
+ .data_len = 4160,
+ .digest = {
+ 0xd1, 0x8e, 0x22, 0xea, 0x44, 0x87, 0x6e, 0x9d,
+ 0xfb, 0x36, 0x02, 0x20, 0x63, 0xb7, 0x69, 0x45,
+ 0x25, 0x41, 0x69, 0xe0, 0x9b, 0x87, 0xcf, 0xa3,
+ 0x51, 0xbb, 0xfc, 0x8d, 0xf7, 0x29, 0xa7, 0xea,
+ },
+ },
+ {
+ .data_len = 4224,
+ .digest = {
+ 0x11, 0x86, 0x7d, 0x84, 0xf9, 0x8c, 0x6e, 0xc4,
+ 0x64, 0x36, 0xc6, 0xf3, 0x42, 0x92, 0x31, 0x2b,
+ 0x1e, 0x12, 0xe6, 0x4d, 0xbe, 0xfa, 0x77, 0x3f,
+ 0x89, 0x41, 0x33, 0x58, 0x1c, 0x98, 0x16, 0x0a,
+ },
+ },
+ {
+ .data_len = 16384,
+ .digest = {
+ 0xb2, 0xba, 0x0c, 0x8c, 0x9d, 0xbb, 0x1e, 0xb0,
+ 0x03, 0xb5, 0xdf, 0x4f, 0xf5, 0x35, 0xdb, 0xec,
+ 0x60, 0xf2, 0x5b, 0xb6, 0xd0, 0x49, 0xd3, 0xed,
+ 0x55, 0xc0, 0x7a, 0xd7, 0xaf, 0xa1, 0xea, 0x53,
+ },
+ },
+};
+
+static const u8 hash_testvec_consolidated[SHA3_256_DIGEST_SIZE] = {
+ 0x3b, 0x33, 0x67, 0xf8, 0xea, 0x92, 0x78, 0x62,
+ 0xdd, 0xbe, 0x72, 0x15, 0xbd, 0x6f, 0xfa, 0xe5,
+ 0x5e, 0xab, 0x9f, 0xb1, 0xe4, 0x23, 0x7c, 0x2c,
+ 0x80, 0xcf, 0x09, 0x75, 0xf8, 0xe2, 0xfa, 0x30,
+};
diff --git a/scripts/crypto/gen-hash-testvecs.py b/scripts/crypto/gen-hash-testvecs.py
index fc063f2ee95f..cb6f6cfbedeb 100755
--- a/scripts/crypto/gen-hash-testvecs.py
+++ b/scripts/crypto/gen-hash-testvecs.py
@@ -61,6 +61,10 @@ def hash_update(ctx, data):
ctx.update(data)
def hash_final(ctx):
+ if ctx.name == "shake_128":
+ return ctx.digest(16)
+ if ctx.name == "shake_256":
+ return ctx.digest(32)
return ctx.digest()
def compute_hash(alg, data):
@@ -87,7 +91,7 @@ def print_c_struct_u8_array_field(name, value):
def alg_digest_size_const(alg):
if alg == 'blake2s':
return 'BLAKE2S_HASH_SIZE'
- return f'{alg.upper()}_DIGEST_SIZE'
+ return f'{alg.upper().replace('-', '_')}_DIGEST_SIZE'
def gen_unkeyed_testvecs(alg):
print('')
@@ -122,7 +126,7 @@ def gen_hmac_testvecs(alg):
ctx.update(mac)
print_static_u8_array_definition(
f'hmac_testvec_consolidated[{alg.upper()}_DIGEST_SIZE]',
- ctx.digest())
+ hash_final(ctx))
BLAKE2S_KEY_SIZE = 32
BLAKE2S_HASH_SIZE = 32
@@ -164,5 +168,7 @@ if alg == 'blake2s':
gen_additional_blake2s_testvecs()
elif alg == 'poly1305':
gen_additional_poly1305_testvecs()
+elif alg.startswith('sha3-') or alg.startswith('shake'):
+ pass # no HMAC
else:
gen_hmac_testvecs(alg)
^ permalink raw reply related
* [PATCH v6 06/17] crypto/jitterentropy: Use lib/crypto/sha3
From: David Howells @ 2025-10-17 14:42 UTC (permalink / raw)
To: Eric Biggers
Cc: David Howells, Jason A . Donenfeld, Ard Biesheuvel, Herbert Xu,
Stephan Mueller, Lukas Wunner, Ignat Korchagin, Luis Chamberlain,
Petr Pavlu, Daniel Gomez, Sami Tolvanen, linux-crypto, keyrings,
linux-modules, linux-kernel
In-Reply-To: <20251017144311.817771-1-dhowells@redhat.com>
Make the jitterentropy RNG use lib/crypto/sha3 rather than crypto/sha3.
For some reason it goes absolutely wild if crypto/sha3 is reimplemented
to use lib/crypto/sha3, but it's fine if it uses lib directly.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Stephan Mueller <smueller@chronox.de>
cc: Herbert Xu <herbert@gondor.apana.org.au>
cc: linux-crypto@vger.kernel.org
---
crypto/jitterentropy-kcapi.c | 100 ++++++++++-------------------------
crypto/jitterentropy.c | 7 +--
crypto/jitterentropy.h | 8 +--
3 files changed, 36 insertions(+), 79 deletions(-)
diff --git a/crypto/jitterentropy-kcapi.c b/crypto/jitterentropy-kcapi.c
index a53de7affe8d..b38118fe51cd 100644
--- a/crypto/jitterentropy-kcapi.c
+++ b/crypto/jitterentropy-kcapi.c
@@ -101,23 +101,13 @@ void jent_get_nstime(__u64 *out)
jent_raw_hires_entropy_store(tmp);
}
-int jent_hash_time(void *hash_state, __u64 time, u8 *addtl,
+int jent_hash_time(struct sha3_256_ctx *hash_state, __u64 time, u8 *addtl,
unsigned int addtl_len, __u64 hash_loop_cnt,
unsigned int stuck)
{
- struct shash_desc *hash_state_desc = (struct shash_desc *)hash_state;
- SHASH_DESC_ON_STACK(desc, hash_state_desc->tfm);
+ struct sha3_256_ctx desc;
u8 intermediary[SHA3_256_DIGEST_SIZE];
__u64 j = 0;
- int ret;
-
- desc->tfm = hash_state_desc->tfm;
-
- if (sizeof(intermediary) != crypto_shash_digestsize(desc->tfm)) {
- pr_warn_ratelimited("Unexpected digest size\n");
- return -EINVAL;
- }
- kmsan_unpoison_memory(intermediary, sizeof(intermediary));
/*
* This loop fills a buffer which is injected into the entropy pool.
@@ -130,24 +120,20 @@ int jent_hash_time(void *hash_state, __u64 time, u8 *addtl,
*
* Note, it does not matter which or how much data you inject, we are
* interested in one Keccack1600 compression operation performed with
- * the crypto_shash_final.
+ * the sha3_256_final.
*/
for (j = 0; j < hash_loop_cnt; j++) {
- ret = crypto_shash_init(desc) ?:
- crypto_shash_update(desc, intermediary,
- sizeof(intermediary)) ?:
- crypto_shash_finup(desc, addtl, addtl_len, intermediary);
- if (ret)
- goto err;
+ sha3_256_init(&desc);
+ sha3_256_update(&desc, intermediary, sizeof(intermediary));
+ sha3_256_update(&desc, addtl, addtl_len);
+ sha3_256_final(&desc, intermediary);
}
/*
* Inject the data from the previous loop into the pool. This data is
* not considered to contain any entropy, but it stirs the pool a bit.
*/
- ret = crypto_shash_update(hash_state_desc, intermediary, sizeof(intermediary));
- if (ret)
- goto err;
+ sha3_256_update(hash_state, intermediary, sizeof(intermediary));
/*
* Insert the time stamp into the hash context representing the pool.
@@ -162,30 +148,25 @@ int jent_hash_time(void *hash_state, __u64 time, u8 *addtl,
time = 0;
}
- ret = crypto_shash_update(hash_state_desc, (u8 *)&time, sizeof(__u64));
-
-err:
- shash_desc_zero(desc);
+ sha3_256_update(hash_state, (u8 *)&time, sizeof(__u64));
memzero_explicit(intermediary, sizeof(intermediary));
-
- return ret;
+ return 0;
}
-int jent_read_random_block(void *hash_state, char *dst, unsigned int dst_len)
+int jent_read_random_block(struct sha3_256_ctx *hash_state, char *dst, unsigned int dst_len)
{
- struct shash_desc *hash_state_desc = (struct shash_desc *)hash_state;
u8 jent_block[SHA3_256_DIGEST_SIZE];
+
/* Obtain data from entropy pool and re-initialize it */
- int ret = crypto_shash_final(hash_state_desc, jent_block) ?:
- crypto_shash_init(hash_state_desc) ?:
- crypto_shash_update(hash_state_desc, jent_block,
- sizeof(jent_block));
+ sha3_256_final(hash_state, jent_block);
+ sha3_256_init(hash_state);
+ sha3_256_update(hash_state, jent_block, sizeof(jent_block));
- if (!ret && dst_len)
+ if (dst_len)
memcpy(dst, jent_block, dst_len);
memzero_explicit(jent_block, sizeof(jent_block));
- return ret;
+ return 0;
}
/***************************************************************************
@@ -195,8 +176,7 @@ int jent_read_random_block(void *hash_state, char *dst, unsigned int dst_len)
struct jitterentropy {
spinlock_t jent_lock;
struct rand_data *entropy_collector;
- struct crypto_shash *tfm;
- struct shash_desc *sdesc;
+ struct sha3_256_ctx *sdesc;
};
static void jent_kcapi_cleanup(struct crypto_tfm *tfm)
@@ -205,16 +185,10 @@ static void jent_kcapi_cleanup(struct crypto_tfm *tfm)
spin_lock(&rng->jent_lock);
- if (rng->sdesc) {
- shash_desc_zero(rng->sdesc);
- kfree(rng->sdesc);
- }
+ if (rng->sdesc)
+ kfree_sensitive(rng->sdesc);
rng->sdesc = NULL;
- if (rng->tfm)
- crypto_free_shash(rng->tfm);
- rng->tfm = NULL;
-
if (rng->entropy_collector)
jent_entropy_collector_free(rng->entropy_collector);
rng->entropy_collector = NULL;
@@ -224,9 +198,8 @@ static void jent_kcapi_cleanup(struct crypto_tfm *tfm)
static int jent_kcapi_init(struct crypto_tfm *tfm)
{
struct jitterentropy *rng = crypto_tfm_ctx(tfm);
- struct crypto_shash *hash;
- struct shash_desc *sdesc;
- int size, ret = 0;
+ struct sha3_256_ctx *sdesc;
+ int ret = 0;
spin_lock_init(&rng->jent_lock);
@@ -239,22 +212,13 @@ static int jent_kcapi_init(struct crypto_tfm *tfm)
* using a fast implementation, we would need to call it more often
* as its variations are lower.
*/
- hash = crypto_alloc_shash(JENT_CONDITIONING_HASH, 0, 0);
- if (IS_ERR(hash)) {
- pr_err("Cannot allocate conditioning digest\n");
- return PTR_ERR(hash);
- }
- rng->tfm = hash;
-
- size = sizeof(struct shash_desc) + crypto_shash_descsize(hash);
- sdesc = kmalloc(size, GFP_KERNEL);
+ sdesc = kmalloc(sizeof(struct sha3_256_ctx), GFP_KERNEL);
if (!sdesc) {
ret = -ENOMEM;
goto err;
}
- sdesc->tfm = hash;
- crypto_shash_init(sdesc);
+ sha3_256_init(sdesc);
rng->sdesc = sdesc;
rng->entropy_collector =
@@ -334,23 +298,15 @@ static struct rng_alg jent_alg = {
static int __init jent_mod_init(void)
{
- SHASH_DESC_ON_STACK(desc, tfm);
- struct crypto_shash *tfm;
+ struct sha3_256_ctx desc;
int ret = 0;
jent_testing_init();
- tfm = crypto_alloc_shash(JENT_CONDITIONING_HASH, 0, 0);
- if (IS_ERR(tfm)) {
- jent_testing_exit();
- return PTR_ERR(tfm);
- }
+ sha3_256_init(&desc);
- desc->tfm = tfm;
- crypto_shash_init(desc);
- ret = jent_entropy_init(CONFIG_CRYPTO_JITTERENTROPY_OSR, 0, desc, NULL);
- shash_desc_zero(desc);
- crypto_free_shash(tfm);
+ ret = jent_entropy_init(CONFIG_CRYPTO_JITTERENTROPY_OSR, 0, &desc, NULL);
+ memzero_explicit(&desc, sizeof(desc));
if (ret) {
/* Handle permanent health test error */
if (fips_enabled)
diff --git a/crypto/jitterentropy.c b/crypto/jitterentropy.c
index 3f93cdc9a7af..36701447cc85 100644
--- a/crypto/jitterentropy.c
+++ b/crypto/jitterentropy.c
@@ -68,7 +68,7 @@ struct rand_data {
* of the RNG are marked as SENSITIVE. A user must not
* access that information while the RNG executes its loops to
* calculate the next random value. */
- void *hash_state; /* SENSITIVE hash state entropy pool */
+ struct sha3_256_ctx *hash_state; /* SENSITIVE hash state entropy pool */
__u64 prev_time; /* SENSITIVE Previous time stamp */
__u64 last_delta; /* SENSITIVE stuck test */
__s64 last_delta2; /* SENSITIVE stuck test */
@@ -656,7 +656,7 @@ int jent_read_entropy(struct rand_data *ec, unsigned char *data,
struct rand_data *jent_entropy_collector_alloc(unsigned int osr,
unsigned int flags,
- void *hash_state)
+ struct sha3_256_ctx *hash_state)
{
struct rand_data *entropy_collector;
@@ -704,7 +704,8 @@ void jent_entropy_collector_free(struct rand_data *entropy_collector)
jent_zfree(entropy_collector);
}
-int jent_entropy_init(unsigned int osr, unsigned int flags, void *hash_state,
+int jent_entropy_init(unsigned int osr, unsigned int flags,
+ struct sha3_256_ctx *hash_state,
struct rand_data *p_ec)
{
/*
diff --git a/crypto/jitterentropy.h b/crypto/jitterentropy.h
index 4c5dbf2a8d8f..e4acbce79d9d 100644
--- a/crypto/jitterentropy.h
+++ b/crypto/jitterentropy.h
@@ -5,20 +5,20 @@ extern void jent_kvzfree(void *ptr, unsigned int len);
extern void *jent_zalloc(unsigned int len);
extern void jent_zfree(void *ptr);
extern void jent_get_nstime(__u64 *out);
-extern int jent_hash_time(void *hash_state, __u64 time, u8 *addtl,
+extern int jent_hash_time(struct sha3_256_ctx *hash_state, __u64 time, u8 *addtl,
unsigned int addtl_len, __u64 hash_loop_cnt,
unsigned int stuck);
-int jent_read_random_block(void *hash_state, char *dst, unsigned int dst_len);
+int jent_read_random_block(struct sha3_256_ctx *hash_state, char *dst, unsigned int dst_len);
struct rand_data;
extern int jent_entropy_init(unsigned int osr, unsigned int flags,
- void *hash_state, struct rand_data *p_ec);
+ struct sha3_256_ctx *hash_state, struct rand_data *p_ec);
extern int jent_read_entropy(struct rand_data *ec, unsigned char *data,
unsigned int len);
extern struct rand_data *jent_entropy_collector_alloc(unsigned int osr,
unsigned int flags,
- void *hash_state);
+ struct sha3_256_ctx *hash_state);
extern void jent_entropy_collector_free(struct rand_data *entropy_collector);
#ifdef CONFIG_CRYPTO_JITTERENTROPY_TESTINTERFACE
^ permalink raw reply related
* [PATCH v6 07/17] crypto/sha3: Use lib/crypto/sha3
From: David Howells @ 2025-10-17 14:42 UTC (permalink / raw)
To: Eric Biggers
Cc: David Howells, Jason A . Donenfeld, Ard Biesheuvel, Herbert Xu,
Stephan Mueller, Lukas Wunner, Ignat Korchagin, Luis Chamberlain,
Petr Pavlu, Daniel Gomez, Sami Tolvanen, linux-crypto, keyrings,
linux-modules, linux-kernel
In-Reply-To: <20251017144311.817771-1-dhowells@redhat.com>
Switch crypto/sha3_generic.c to use lib/crypto/sha3. Note that this makes
use of the internal general API rather implementing a separate set of
init/update/finup handlers for each algorithm.
Whilst we're at it, eliminate the sha3_state struct and move the st[] state
array into the sha3_ctx struct.
Note that this also eliminates the need to set CRYPTO_AHASH_ALG_BLOCK_ONLY
as the lib/crypto/sha3.c code handles misalignment itself without the need
for a place to stash a block at the cost of a few extra bytes of state.
This may mean that HASH_MAX_STATESIZE can be reduced.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Eric Biggers <ebiggers@kernel.org>
cc: Jason A. Donenfeld <Jason@zx2c4.com>
cc: Ard Biesheuvel <ardb@kernel.org>
cc: Herbert Xu <herbert@gondor.apana.org.au>
cc: Stephan Mueller <smueller@chronox.de>
cc: linux-crypto@vger.kernel.org
---
Documentation/crypto/sha3.rst | 8 +-
arch/arm64/crypto/sha3-ce-glue.c | 25 ++--
crypto/sha3_generic.c | 209 ++++---------------------------
include/crypto/sha3.h | 6 +-
lib/crypto/sha3.c | 35 +++---
5 files changed, 56 insertions(+), 227 deletions(-)
diff --git a/Documentation/crypto/sha3.rst b/Documentation/crypto/sha3.rst
index 3255618ea2ec..69d375567b94 100644
--- a/Documentation/crypto/sha3.rst
+++ b/Documentation/crypto/sha3.rst
@@ -151,7 +151,7 @@ further algorithms or APIs as the engine in the same in all cases. The
algorithm APIs all wrap the common context structure::
struct sha3_ctx {
- struct sha3_state state;
+ u64 st[SHA3_STATE_SIZE / 8];
u8 block_size;
u8 padding;
u8 absorb_offset;
@@ -159,13 +159,9 @@ algorithm APIs all wrap the common context structure::
bool end_marked;
};
- struct sha3_state {
- u64 st[SHA3_STATE_SIZE / 8];
- };
-
The fields are as follows:
- * ``state.st``
+ * ``st``
An array of 25 64-bit state buckets that are used to hold the mathematical
state of the Keccak engine. Data is XOR'd onto part of this, the engine is
diff --git a/arch/arm64/crypto/sha3-ce-glue.c b/arch/arm64/crypto/sha3-ce-glue.c
index 426d8044535a..1f05cc328e11 100644
--- a/arch/arm64/crypto/sha3-ce-glue.c
+++ b/arch/arm64/crypto/sha3-ce-glue.c
@@ -28,13 +28,18 @@ MODULE_ALIAS_CRYPTO("sha3-256");
MODULE_ALIAS_CRYPTO("sha3-384");
MODULE_ALIAS_CRYPTO("sha3-512");
+static struct sha3_ctx *crypto_sha3_desc(struct shash_desc *desc)
+{
+ return shash_desc_ctx(desc);
+}
+
asmlinkage int sha3_ce_transform(u64 *st, const u8 *data, int blocks,
int md_len);
static int arm64_sha3_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
- struct sha3_state *sctx = shash_desc_ctx(desc);
+ struct sha3_ctx *ctx = crypto_sha3_desc(desc);
struct crypto_shash *tfm = desc->tfm;
unsigned int bs, ds;
int blocks;
@@ -47,7 +52,7 @@ static int arm64_sha3_update(struct shash_desc *desc, const u8 *data,
int rem;
kernel_neon_begin();
- rem = sha3_ce_transform(sctx->st, data, blocks, ds);
+ rem = sha3_ce_transform(ctx->st, data, blocks, ds);
kernel_neon_end();
data += (blocks - rem) * bs;
blocks = rem;
@@ -58,7 +63,7 @@ static int arm64_sha3_update(struct shash_desc *desc, const u8 *data,
static int arm64_sha3_finup(struct shash_desc *desc, const u8 *src, unsigned int len,
u8 *out)
{
- struct sha3_state *sctx = shash_desc_ctx(desc);
+ struct sha3_ctx *ctx = crypto_sha3_desc(desc);
struct crypto_shash *tfm = desc->tfm;
__le64 *digest = (__le64 *)out;
u8 block[SHA3_224_BLOCK_SIZE];
@@ -74,15 +79,15 @@ static int arm64_sha3_finup(struct shash_desc *desc, const u8 *src, unsigned int
block[bs - 1] |= 0x80;
kernel_neon_begin();
- sha3_ce_transform(sctx->st, block, 1, ds);
+ sha3_ce_transform(ctx->st, block, 1, ds);
kernel_neon_end();
memzero_explicit(block , sizeof(block));
for (i = 0; i < ds / 8; i++)
- put_unaligned_le64(sctx->st[i], digest++);
+ put_unaligned_le64(ctx->st[i], digest++);
if (ds & 4)
- put_unaligned_le32(sctx->st[i], (__le32 *)digest);
+ put_unaligned_le32(ctx->st[i], (__le32 *)digest);
return 0;
}
@@ -92,7 +97,7 @@ static struct shash_alg algs[] = { {
.init = crypto_sha3_init,
.update = arm64_sha3_update,
.finup = arm64_sha3_finup,
- .descsize = SHA3_STATE_SIZE,
+ .descsize = sizeof(struct sha3_ctx),
.base.cra_name = "sha3-224",
.base.cra_driver_name = "sha3-224-ce",
.base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
@@ -104,7 +109,7 @@ static struct shash_alg algs[] = { {
.init = crypto_sha3_init,
.update = arm64_sha3_update,
.finup = arm64_sha3_finup,
- .descsize = SHA3_STATE_SIZE,
+ .descsize = sizeof(struct sha3_ctx),
.base.cra_name = "sha3-256",
.base.cra_driver_name = "sha3-256-ce",
.base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
@@ -116,7 +121,7 @@ static struct shash_alg algs[] = { {
.init = crypto_sha3_init,
.update = arm64_sha3_update,
.finup = arm64_sha3_finup,
- .descsize = SHA3_STATE_SIZE,
+ .descsize = sizeof(struct sha3_ctx),
.base.cra_name = "sha3-384",
.base.cra_driver_name = "sha3-384-ce",
.base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
@@ -128,7 +133,7 @@ static struct shash_alg algs[] = { {
.init = crypto_sha3_init,
.update = arm64_sha3_update,
.finup = arm64_sha3_finup,
- .descsize = SHA3_STATE_SIZE,
+ .descsize = sizeof(struct sha3_ctx),
.base.cra_name = "sha3-512",
.base.cra_driver_name = "sha3-512-ce",
.base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
diff --git a/crypto/sha3_generic.c b/crypto/sha3_generic.c
index 41d1e506e6de..7b16e4c27fc8 100644
--- a/crypto/sha3_generic.c
+++ b/crypto/sha3_generic.c
@@ -13,156 +13,19 @@
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/string.h>
-#include <linux/unaligned.h>
-/*
- * On some 32-bit architectures (h8300), GCC ends up using
- * over 1 KB of stack if we inline the round calculation into the loop
- * in keccakf(). On the other hand, on 64-bit architectures with plenty
- * of [64-bit wide] general purpose registers, not inlining it severely
- * hurts performance. So let's use 64-bitness as a heuristic to decide
- * whether to inline or not.
- */
-#ifdef CONFIG_64BIT
-#define SHA3_INLINE inline
-#else
-#define SHA3_INLINE noinline
-#endif
-
-#define KECCAK_ROUNDS 24
-
-static const u64 keccakf_rndc[24] = {
- 0x0000000000000001ULL, 0x0000000000008082ULL, 0x800000000000808aULL,
- 0x8000000080008000ULL, 0x000000000000808bULL, 0x0000000080000001ULL,
- 0x8000000080008081ULL, 0x8000000000008009ULL, 0x000000000000008aULL,
- 0x0000000000000088ULL, 0x0000000080008009ULL, 0x000000008000000aULL,
- 0x000000008000808bULL, 0x800000000000008bULL, 0x8000000000008089ULL,
- 0x8000000000008003ULL, 0x8000000000008002ULL, 0x8000000000000080ULL,
- 0x000000000000800aULL, 0x800000008000000aULL, 0x8000000080008081ULL,
- 0x8000000000008080ULL, 0x0000000080000001ULL, 0x8000000080008008ULL
-};
-
-/* update the state with given number of rounds */
-
-static SHA3_INLINE void keccakf_round(u64 st[25])
-{
- u64 t[5], tt, bc[5];
-
- /* Theta */
- bc[0] = st[0] ^ st[5] ^ st[10] ^ st[15] ^ st[20];
- bc[1] = st[1] ^ st[6] ^ st[11] ^ st[16] ^ st[21];
- bc[2] = st[2] ^ st[7] ^ st[12] ^ st[17] ^ st[22];
- bc[3] = st[3] ^ st[8] ^ st[13] ^ st[18] ^ st[23];
- bc[4] = st[4] ^ st[9] ^ st[14] ^ st[19] ^ st[24];
-
- t[0] = bc[4] ^ rol64(bc[1], 1);
- t[1] = bc[0] ^ rol64(bc[2], 1);
- t[2] = bc[1] ^ rol64(bc[3], 1);
- t[3] = bc[2] ^ rol64(bc[4], 1);
- t[4] = bc[3] ^ rol64(bc[0], 1);
-
- st[0] ^= t[0];
-
- /* Rho Pi */
- tt = st[1];
- st[ 1] = rol64(st[ 6] ^ t[1], 44);
- st[ 6] = rol64(st[ 9] ^ t[4], 20);
- st[ 9] = rol64(st[22] ^ t[2], 61);
- st[22] = rol64(st[14] ^ t[4], 39);
- st[14] = rol64(st[20] ^ t[0], 18);
- st[20] = rol64(st[ 2] ^ t[2], 62);
- st[ 2] = rol64(st[12] ^ t[2], 43);
- st[12] = rol64(st[13] ^ t[3], 25);
- st[13] = rol64(st[19] ^ t[4], 8);
- st[19] = rol64(st[23] ^ t[3], 56);
- st[23] = rol64(st[15] ^ t[0], 41);
- st[15] = rol64(st[ 4] ^ t[4], 27);
- st[ 4] = rol64(st[24] ^ t[4], 14);
- st[24] = rol64(st[21] ^ t[1], 2);
- st[21] = rol64(st[ 8] ^ t[3], 55);
- st[ 8] = rol64(st[16] ^ t[1], 45);
- st[16] = rol64(st[ 5] ^ t[0], 36);
- st[ 5] = rol64(st[ 3] ^ t[3], 28);
- st[ 3] = rol64(st[18] ^ t[3], 21);
- st[18] = rol64(st[17] ^ t[2], 15);
- st[17] = rol64(st[11] ^ t[1], 10);
- st[11] = rol64(st[ 7] ^ t[2], 6);
- st[ 7] = rol64(st[10] ^ t[0], 3);
- st[10] = rol64( tt ^ t[1], 1);
-
- /* Chi */
- bc[ 0] = ~st[ 1] & st[ 2];
- bc[ 1] = ~st[ 2] & st[ 3];
- bc[ 2] = ~st[ 3] & st[ 4];
- bc[ 3] = ~st[ 4] & st[ 0];
- bc[ 4] = ~st[ 0] & st[ 1];
- st[ 0] ^= bc[ 0];
- st[ 1] ^= bc[ 1];
- st[ 2] ^= bc[ 2];
- st[ 3] ^= bc[ 3];
- st[ 4] ^= bc[ 4];
-
- bc[ 0] = ~st[ 6] & st[ 7];
- bc[ 1] = ~st[ 7] & st[ 8];
- bc[ 2] = ~st[ 8] & st[ 9];
- bc[ 3] = ~st[ 9] & st[ 5];
- bc[ 4] = ~st[ 5] & st[ 6];
- st[ 5] ^= bc[ 0];
- st[ 6] ^= bc[ 1];
- st[ 7] ^= bc[ 2];
- st[ 8] ^= bc[ 3];
- st[ 9] ^= bc[ 4];
-
- bc[ 0] = ~st[11] & st[12];
- bc[ 1] = ~st[12] & st[13];
- bc[ 2] = ~st[13] & st[14];
- bc[ 3] = ~st[14] & st[10];
- bc[ 4] = ~st[10] & st[11];
- st[10] ^= bc[ 0];
- st[11] ^= bc[ 1];
- st[12] ^= bc[ 2];
- st[13] ^= bc[ 3];
- st[14] ^= bc[ 4];
-
- bc[ 0] = ~st[16] & st[17];
- bc[ 1] = ~st[17] & st[18];
- bc[ 2] = ~st[18] & st[19];
- bc[ 3] = ~st[19] & st[15];
- bc[ 4] = ~st[15] & st[16];
- st[15] ^= bc[ 0];
- st[16] ^= bc[ 1];
- st[17] ^= bc[ 2];
- st[18] ^= bc[ 3];
- st[19] ^= bc[ 4];
-
- bc[ 0] = ~st[21] & st[22];
- bc[ 1] = ~st[22] & st[23];
- bc[ 2] = ~st[23] & st[24];
- bc[ 3] = ~st[24] & st[20];
- bc[ 4] = ~st[20] & st[21];
- st[20] ^= bc[ 0];
- st[21] ^= bc[ 1];
- st[22] ^= bc[ 2];
- st[23] ^= bc[ 3];
- st[24] ^= bc[ 4];
-}
-
-static void keccakf(u64 st[25])
+static struct sha3_ctx *crypto_sha3_desc(struct shash_desc *desc)
{
- int round;
-
- for (round = 0; round < KECCAK_ROUNDS; round++) {
- keccakf_round(st);
- /* Iota */
- st[0] ^= keccakf_rndc[round];
- }
+ return shash_desc_ctx(desc);
}
int crypto_sha3_init(struct shash_desc *desc)
{
- struct sha3_state *sctx = shash_desc_ctx(desc);
+ struct sha3_ctx *ctx = crypto_sha3_desc(desc);
- memset(sctx->st, 0, sizeof(sctx->st));
+ memset(ctx, 0, sizeof(*ctx));
+ ctx->block_size = crypto_shash_blocksize(desc->tfm);
+ ctx->padding = 0x06;
return 0;
}
EXPORT_SYMBOL(crypto_sha3_init);
@@ -170,51 +33,21 @@ EXPORT_SYMBOL(crypto_sha3_init);
static int crypto_sha3_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
- unsigned int rsiz = crypto_shash_blocksize(desc->tfm);
- struct sha3_state *sctx = shash_desc_ctx(desc);
- unsigned int rsizw = rsiz / 8;
-
- do {
- int i;
+ struct sha3_ctx *ctx = crypto_sha3_desc(desc);
- for (i = 0; i < rsizw; i++)
- sctx->st[i] ^= get_unaligned_le64(data + 8 * i);
- keccakf(sctx->st);
-
- data += rsiz;
- len -= rsiz;
- } while (len >= rsiz);
+ sha3_update(ctx, data, len);
return len;
}
static int crypto_sha3_finup(struct shash_desc *desc, const u8 *src,
unsigned int len, u8 *out)
{
- unsigned int digest_size = crypto_shash_digestsize(desc->tfm);
- unsigned int rsiz = crypto_shash_blocksize(desc->tfm);
- struct sha3_state *sctx = shash_desc_ctx(desc);
- __le64 block[SHA3_224_BLOCK_SIZE / 8] = {};
- __le64 *digest = (__le64 *)out;
- unsigned int rsizw = rsiz / 8;
- u8 *p;
- int i;
-
- p = memcpy(block, src, len);
- p[len++] = 0x06;
- p[rsiz - 1] |= 0x80;
-
- for (i = 0; i < rsizw; i++)
- sctx->st[i] ^= le64_to_cpu(block[i]);
- memzero_explicit(block, sizeof(block));
-
- keccakf(sctx->st);
-
- for (i = 0; i < digest_size / 8; i++)
- put_unaligned_le64(sctx->st[i], digest++);
-
- if (digest_size & 4)
- put_unaligned_le32(sctx->st[i], (__le32 *)digest);
+ struct sha3_ctx *ctx = crypto_sha3_desc(desc);
+ if (len && src)
+ sha3_update(ctx, src, len);
+ sha3_squeeze(ctx, out, crypto_shash_digestsize(desc->tfm));
+ sha3_clear(ctx);
return 0;
}
@@ -223,10 +56,10 @@ static struct shash_alg algs[] = { {
.init = crypto_sha3_init,
.update = crypto_sha3_update,
.finup = crypto_sha3_finup,
- .descsize = SHA3_STATE_SIZE,
+ .descsize = sizeof(struct sha3_ctx),
.base.cra_name = "sha3-224",
.base.cra_driver_name = "sha3-224-generic",
- .base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
+ .base.cra_flags = 0,
.base.cra_blocksize = SHA3_224_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
}, {
@@ -234,10 +67,10 @@ static struct shash_alg algs[] = { {
.init = crypto_sha3_init,
.update = crypto_sha3_update,
.finup = crypto_sha3_finup,
- .descsize = SHA3_STATE_SIZE,
+ .descsize = sizeof(struct sha3_ctx),
.base.cra_name = "sha3-256",
.base.cra_driver_name = "sha3-256-generic",
- .base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
+ .base.cra_flags = 0,
.base.cra_blocksize = SHA3_256_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
}, {
@@ -245,10 +78,10 @@ static struct shash_alg algs[] = { {
.init = crypto_sha3_init,
.update = crypto_sha3_update,
.finup = crypto_sha3_finup,
- .descsize = SHA3_STATE_SIZE,
+ .descsize = sizeof(struct sha3_ctx),
.base.cra_name = "sha3-384",
.base.cra_driver_name = "sha3-384-generic",
- .base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
+ .base.cra_flags = 0,
.base.cra_blocksize = SHA3_384_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
}, {
@@ -256,10 +89,10 @@ static struct shash_alg algs[] = { {
.init = crypto_sha3_init,
.update = crypto_sha3_update,
.finup = crypto_sha3_finup,
- .descsize = SHA3_STATE_SIZE,
+ .descsize = sizeof(struct sha3_ctx),
.base.cra_name = "sha3-512",
.base.cra_driver_name = "sha3-512-generic",
- .base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
+ .base.cra_flags = 0,
.base.cra_blocksize = SHA3_512_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
} };
diff --git a/include/crypto/sha3.h b/include/crypto/sha3.h
index a54117cb1546..7e845dde55ac 100644
--- a/include/crypto/sha3.h
+++ b/include/crypto/sha3.h
@@ -38,10 +38,6 @@
struct shash_desc;
-struct sha3_state {
- u64 st[SHA3_STATE_SIZE / 8];
-};
-
/*
* The SHA3 context structure and state buffer.
*
@@ -52,7 +48,7 @@ struct sha3_state {
* byteswap step is a no-op.
*/
struct sha3_ctx {
- struct sha3_state state;
+ u64 st[SHA3_STATE_SIZE / 8];
u8 block_size; /* Block size in bytes */
u8 padding; /* Padding byte */
u8 absorb_offset; /* Next state byte to absorb into */
diff --git a/lib/crypto/sha3.c b/lib/crypto/sha3.c
index 2c292b0b3db3..911b2c35dbc4 100644
--- a/lib/crypto/sha3.c
+++ b/lib/crypto/sha3.c
@@ -48,10 +48,10 @@ static const u64 sha3_keccakf_rndc[24] = {
/*
* Perform a single round of Keccak mixing.
*/
-static SHA3_INLINE void sha3_keccakf_one_round_generic(struct sha3_state *state,
+static SHA3_INLINE void sha3_keccakf_one_round_generic(struct sha3_ctx *ctx,
int round)
{
- u64 *st = state->st;
+ u64 *st = ctx->st;
u64 t[5], tt, bc[5];
/* Theta */
@@ -153,13 +153,13 @@ static SHA3_INLINE void sha3_keccakf_one_round_generic(struct sha3_state *state,
st[24] ^= bc[ 4];
/* Iota */
- state->st[0] ^= sha3_keccakf_rndc[round];
+ ctx->st[0] ^= sha3_keccakf_rndc[round];
}
-static void sha3_keccakf_rounds_generic(struct sha3_state *state)
+static void sha3_keccakf_rounds_generic(struct sha3_ctx *ctx)
{
for (int round = 0; round < SHA3_KECCAK_ROUNDS; round++)
- sha3_keccakf_one_round_generic(state, round);
+ sha3_keccakf_one_round_generic(ctx, round);
}
/*
@@ -167,25 +167,24 @@ static void sha3_keccakf_rounds_generic(struct sha3_state *state)
* machine for the duration of the Keccak mixing function. Note that these
* loops are no-ops on LE machines and will be optimised away.
*/
-static void sha3_keccakf_generic(struct sha3_state *state)
+static void sha3_keccakf_generic(struct sha3_ctx *ctx)
{
- for (int i = 0; i < ARRAY_SIZE(state->st); i++)
- cpu_to_le64s(&state->st[i]);
+ for (int i = 0; i < ARRAY_SIZE(ctx->st); i++)
+ cpu_to_le64s(&ctx->st[i]);
- sha3_keccakf_rounds_generic(state);
+ sha3_keccakf_rounds_generic(ctx);
- for (int i = 0; i < ARRAY_SIZE(state->st); i++)
- le64_to_cpus(&state->st[i]);
+ for (int i = 0; i < ARRAY_SIZE(ctx->st); i++)
+ le64_to_cpus(&ctx->st[i]);
}
static void sha3_absorb_block_generic(struct sha3_ctx *ctx, const u8 *data)
{
- struct sha3_state *state = &ctx->state;
size_t bsize = ctx->block_size;
for (size_t i = 0; i < bsize / 8; i++)
- state->st[i] ^= get_unaligned((u64 *)(data + 8 * i));
- sha3_keccakf_generic(state);
+ ctx->st[i] ^= get_unaligned((u64 *)(data + 8 * i));
+ sha3_keccakf_generic(ctx);
}
/*
@@ -214,7 +213,7 @@ static void sha3_absorb_blocks_generic(struct sha3_ctx *ctx,
static void sha3_absorb_xorle(struct sha3_ctx *ctx, const u8 *data,
size_t partial, size_t len)
{
- u8 *buf = (u8 *)ctx->state.st;
+ u8 *buf = (u8 *)ctx->st;
buf += partial;
for (size_t i = 0; i < len; i++)
@@ -243,7 +242,7 @@ void sha3_update(struct sha3_ctx *ctx, const u8 *data, size_t len)
sha3_absorb_xorle(ctx, data, absorb_offset, bsize - absorb_offset);
len -= bsize - absorb_offset;
data += bsize - absorb_offset;
- sha3_keccakf(&ctx->state);
+ sha3_keccakf(ctx);
ctx->absorb_offset = 0;
}
@@ -282,7 +281,7 @@ void sha3_squeeze(struct sha3_ctx *ctx, u8 *out, size_t out_size)
{
size_t squeeze_offset = ctx->squeeze_offset;
size_t bsize = ctx->block_size;
- u8 *p = (u8 *)ctx->state.st, end_marker = 0x80;
+ u8 *p = (u8 *)ctx->st, end_marker = 0x80;
if (!ctx->end_marked) {
sha3_absorb_xorle(ctx, &ctx->padding, ctx->absorb_offset, 1);
@@ -292,7 +291,7 @@ void sha3_squeeze(struct sha3_ctx *ctx, u8 *out, size_t out_size)
for (;;) {
if (squeeze_offset == 0)
- sha3_keccakf(&ctx->state);
+ sha3_keccakf(ctx);
size_t part = umin(out_size, bsize - squeeze_offset);
^ permalink raw reply related
* [PATCH v6 08/17] crypto/sha3: Add SHAKE128/256 support
From: David Howells @ 2025-10-17 14:42 UTC (permalink / raw)
To: Eric Biggers
Cc: David Howells, Jason A . Donenfeld, Ard Biesheuvel, Herbert Xu,
Stephan Mueller, Lukas Wunner, Ignat Korchagin, Luis Chamberlain,
Petr Pavlu, Daniel Gomez, Sami Tolvanen, linux-crypto, keyrings,
linux-modules, linux-kernel
In-Reply-To: <20251017144311.817771-1-dhowells@redhat.com>
SHAKE128/256 'digest' algos need to be available for the ML-DSA pre-digest,
which is a selectable algorithm and need to be available through the same
API as, say, SHA3-512 and SHA512 both. Resqueezability (probably) isn't
required for this and they'll produce the default number of bytes as the
digest size.
Also, increase MAX_ALGAPI_BLOCKSIZE to accommodate SHAKE128.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Eric Biggers <ebiggers@kernel.org>
cc: Jason A. Donenfeld <Jason@zx2c4.com>
cc: Ard Biesheuvel <ardb@kernel.org>
cc: Herbert Xu <herbert@gondor.apana.org.au>
cc: Stephan Mueller <smueller@chronox.de>
cc: linux-crypto@vger.kernel.org
---
crypto/sha3_generic.c | 26 ++++++++++++++++++++++++++
include/crypto/algapi.h | 2 +-
2 files changed, 27 insertions(+), 1 deletion(-)
diff --git a/crypto/sha3_generic.c b/crypto/sha3_generic.c
index 7b16e4c27fc8..3e5a23022596 100644
--- a/crypto/sha3_generic.c
+++ b/crypto/sha3_generic.c
@@ -95,6 +95,28 @@ static struct shash_alg algs[] = { {
.base.cra_flags = 0,
.base.cra_blocksize = SHA3_512_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
+}, {
+ .digestsize = SHAKE128_DEFAULT_SIZE,
+ .init = crypto_sha3_init,
+ .update = crypto_sha3_update,
+ .finup = crypto_sha3_finup,
+ .descsize = sizeof(struct sha3_ctx),
+ .base.cra_name = "shake128",
+ .base.cra_driver_name = "shake128-generic",
+ .base.cra_flags = 0,
+ .base.cra_blocksize = SHAKE128_BLOCK_SIZE,
+ .base.cra_module = THIS_MODULE,
+}, {
+ .digestsize = SHAKE256_DEFAULT_SIZE,
+ .init = crypto_sha3_init,
+ .update = crypto_sha3_update,
+ .finup = crypto_sha3_finup,
+ .descsize = sizeof(struct sha3_ctx),
+ .base.cra_name = "shake256",
+ .base.cra_driver_name = "shake256-generic",
+ .base.cra_flags = 0,
+ .base.cra_blocksize = SHAKE256_BLOCK_SIZE,
+ .base.cra_module = THIS_MODULE,
} };
static int __init sha3_generic_mod_init(void)
@@ -121,3 +143,7 @@ MODULE_ALIAS_CRYPTO("sha3-384");
MODULE_ALIAS_CRYPTO("sha3-384-generic");
MODULE_ALIAS_CRYPTO("sha3-512");
MODULE_ALIAS_CRYPTO("sha3-512-generic");
+MODULE_ALIAS_CRYPTO("shake128");
+MODULE_ALIAS_CRYPTO("shake128-generic");
+MODULE_ALIAS_CRYPTO("shake256");
+MODULE_ALIAS_CRYPTO("shake256-generic");
diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h
index fc4574940636..6209762736c4 100644
--- a/include/crypto/algapi.h
+++ b/include/crypto/algapi.h
@@ -20,7 +20,7 @@
* static buffers that are big enough for any combination of
* algs and architectures. Ciphers have a lower maximum size.
*/
-#define MAX_ALGAPI_BLOCKSIZE 160
+#define MAX_ALGAPI_BLOCKSIZE 168
#define MAX_ALGAPI_ALIGNMASK 127
#define MAX_CIPHER_BLOCKSIZE 16
#define MAX_CIPHER_ALIGNMASK 15
^ permalink raw reply related
* [PATCH v6 09/17] crypto: SHAKE tests
From: David Howells @ 2025-10-17 14:42 UTC (permalink / raw)
To: Eric Biggers
Cc: David Howells, Jason A . Donenfeld, Ard Biesheuvel, Herbert Xu,
Stephan Mueller, Lukas Wunner, Ignat Korchagin, Luis Chamberlain,
Petr Pavlu, Daniel Gomez, Sami Tolvanen, linux-crypto, keyrings,
linux-modules, linux-kernel
In-Reply-To: <20251017144311.817771-1-dhowells@redhat.com>
---
crypto/sha3_generic.c | 14 ++++++++--
crypto/testmgr.c | 14 ++++++++++
crypto/testmgr.h | 59 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 85 insertions(+), 2 deletions(-)
diff --git a/crypto/sha3_generic.c b/crypto/sha3_generic.c
index 3e5a23022596..3a338fb704c2 100644
--- a/crypto/sha3_generic.c
+++ b/crypto/sha3_generic.c
@@ -30,6 +30,16 @@ int crypto_sha3_init(struct shash_desc *desc)
}
EXPORT_SYMBOL(crypto_sha3_init);
+static int crypto_shake_init(struct shash_desc *desc)
+{
+ struct sha3_ctx *ctx = crypto_sha3_desc(desc);
+
+ memset(ctx, 0, sizeof(*ctx));
+ ctx->block_size = crypto_shash_blocksize(desc->tfm);
+ ctx->padding = 0x1f;
+ return 0;
+}
+
static int crypto_sha3_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
@@ -97,7 +107,7 @@ static struct shash_alg algs[] = { {
.base.cra_module = THIS_MODULE,
}, {
.digestsize = SHAKE128_DEFAULT_SIZE,
- .init = crypto_sha3_init,
+ .init = crypto_shake_init,
.update = crypto_sha3_update,
.finup = crypto_sha3_finup,
.descsize = sizeof(struct sha3_ctx),
@@ -108,7 +118,7 @@ static struct shash_alg algs[] = { {
.base.cra_module = THIS_MODULE,
}, {
.digestsize = SHAKE256_DEFAULT_SIZE,
- .init = crypto_sha3_init,
+ .init = crypto_shake_init,
.update = crypto_sha3_update,
.finup = crypto_sha3_finup,
.descsize = sizeof(struct sha3_ctx),
diff --git a/crypto/testmgr.c b/crypto/testmgr.c
index 6a490aaa71b9..88d412ceb6f2 100644
--- a/crypto/testmgr.c
+++ b/crypto/testmgr.c
@@ -5516,6 +5516,20 @@ static const struct alg_test_desc alg_test_descs[] = {
.suite = {
.hash = __VECS(sha512_tv_template)
}
+ }, {
+ .alg = "shake128",
+ .test = alg_test_hash,
+ .fips_allowed = 1,
+ .suite = {
+ .hash = __VECS(shake128_tv_template)
+ }
+ }, {
+ .alg = "shake256",
+ .test = alg_test_hash,
+ .fips_allowed = 1,
+ .suite = {
+ .hash = __VECS(shake256_tv_template)
+ }
}, {
.alg = "sm3",
.test = alg_test_hash,
diff --git a/crypto/testmgr.h b/crypto/testmgr.h
index 268231227282..c0eca7f72aa0 100644
--- a/crypto/testmgr.h
+++ b/crypto/testmgr.h
@@ -4690,6 +4690,65 @@ static const struct hash_testvec sha3_512_tv_template[] = {
};
+/* From: https://csrc.nist.gov/projects/cryptographic-standards-and-guidelines/example-values */
+static const struct hash_testvec shake128_tv_template[] = {
+ {
+ .plaintext = "",
+ .psize = 0,
+ .digest =
+ "\x7f\x9c\x2b\xa4\xe8\x8f\x82\x7d\x61\x60\x45\x50\x76\x05\x85\x3e",
+ }, {
+ .plaintext =
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3",
+ .psize = 200,
+ .digest =
+ "\x13\x1a\xb8\xd2\xb5\x94\x94\x6b\x9c\x81\x33\x3f\x9b\xb6\xe0\xce",
+ }
+};
+
+/* From: https://csrc.nist.gov/projects/cryptographic-standards-and-guidelines/example-values */
+static const struct hash_testvec shake256_tv_template[] = {
+ {
+ .plaintext = "",
+ .psize = 0,
+ .digest =
+ "\x46\xb9\xdd\x2b\x0b\xa8\x8d\x13\x23\x3b\x3f\xeb\x74\x3e\xeb\x24"
+ "\x3f\xcd\x52\xea\x62\xb8\x1b\x82\xb5\x0c\x27\x64\x6e\xd5\x76\x2f"
+ }, {
+ .plaintext =
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3"
+ "\xa3\xa3\xa3\xa3\xa3\xa3\xa3\xa3",
+ .psize = 200,
+ .digest =
+ "\xcd\x8a\x92\x0e\xd1\x41\xaa\x04\x07\xa2\x2d\x59\x28\x86\x52\xe9"
+ "\xd9\xf1\xa7\xee\x0c\x1e\x7c\x1c\xa6\x99\x42\x4d\xa8\x4a\x90\x4d",
+ }
+};
+
+
/*
* MD5 test vectors from RFC1321
*/
^ permalink raw reply related
* [PATCH v6 10/17] crypto: Add ML-DSA/Dilithium support
From: David Howells @ 2025-10-17 14:42 UTC (permalink / raw)
To: Eric Biggers
Cc: David Howells, Jason A . Donenfeld, Ard Biesheuvel, Herbert Xu,
Stephan Mueller, Lukas Wunner, Ignat Korchagin, Luis Chamberlain,
Petr Pavlu, Daniel Gomez, Sami Tolvanen, linux-crypto, keyrings,
linux-modules, linux-kernel
In-Reply-To: <20251017144311.817771-1-dhowells@redhat.com>
Port Stephan Mueller's Leancrypto implementation of ML-DSA/Dilithium to the
kernel.
[!] NOTE: This isn't entirely cleaned up yet! This includes converting the
documentation comments and removing more compatibility macros. I wanted to
concentrate on getting it actually working first.
Apologies to Stephan, but I've stripped out a bunch of macros to do return
checking and suchlike and removed a bunch of "lc_" prefixes from the code.
The code retains the following features, some of which probably need to
be separated out or dropped entirely:
- Signature verification
- Signing
- Prehash support
- External Mu support
Composite signature support is mostly removed and none of the arch-specific
code from Leancrypto has been included for the moment, so this is pure C.
The interface to this code is through the crypto_sig API as the PKCS#7 code
wants to use that rather than calling it directly. As such, I've placed it
in crypto/ rather than lib/crypto/.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Stephan Mueller <smueller@chronox.de>
cc: Eric Biggers <ebiggers@kernel.org>
cc: Jason A. Donenfeld <Jason@zx2c4.com>
cc: Ard Biesheuvel <ardb@kernel.org>
cc: Herbert Xu <herbert@gondor.apana.org.au>
cc: linux-crypto@vger.kernel.org
---
crypto/Kconfig | 1 +
crypto/Makefile | 1 +
crypto/ml_dsa/Kconfig | 17 +
crypto/ml_dsa/Makefile | 18 +
crypto/ml_dsa/dilithium.h | 750 ++++++++++++++++++
crypto/ml_dsa/dilithium_44.c | 33 +
crypto/ml_dsa/dilithium_44.h | 377 +++++++++
crypto/ml_dsa/dilithium_65.c | 33 +
crypto/ml_dsa/dilithium_65.h | 377 +++++++++
crypto/ml_dsa/dilithium_87.c | 33 +
crypto/ml_dsa/dilithium_87.h | 377 +++++++++
crypto/ml_dsa/dilithium_api.c | 729 +++++++++++++++++
crypto/ml_dsa/dilithium_debug.h | 80 ++
crypto/ml_dsa/dilithium_ntt.c | 89 +++
crypto/ml_dsa/dilithium_ntt.h | 35 +
crypto/ml_dsa/dilithium_pack.h | 210 +++++
crypto/ml_dsa/dilithium_poly.c | 586 ++++++++++++++
crypto/ml_dsa/dilithium_poly.h | 190 +++++
crypto/ml_dsa/dilithium_poly_c.h | 149 ++++
crypto/ml_dsa/dilithium_poly_common.h | 35 +
crypto/ml_dsa/dilithium_polyvec.h | 343 ++++++++
crypto/ml_dsa/dilithium_polyvec_c.h | 94 +++
crypto/ml_dsa/dilithium_reduce.h | 108 +++
crypto/ml_dsa/dilithium_rounding.c | 128 +++
crypto/ml_dsa/dilithium_rounding.h | 45 ++
crypto/ml_dsa/dilithium_service_helpers.h | 99 +++
crypto/ml_dsa/dilithium_sig.c | 404 ++++++++++
crypto/ml_dsa/dilithium_signature_c.c | 174 ++++
crypto/ml_dsa/dilithium_signature_c.h | 53 ++
crypto/ml_dsa/dilithium_signature_helper.c | 110 +++
crypto/ml_dsa/dilithium_signature_impl.h | 838 ++++++++++++++++++++
crypto/ml_dsa/dilithium_type.h | 259 ++++++
crypto/ml_dsa/dilithium_zetas.c | 67 ++
crypto/ml_dsa/fips_mode.h | 45 ++
crypto/ml_dsa/signature_domain_separation.c | 213 +++++
crypto/ml_dsa/signature_domain_separation.h | 33 +
crypto/ml_dsa/small_stack_support.h | 40 +
37 files changed, 7173 insertions(+)
create mode 100644 crypto/ml_dsa/Kconfig
create mode 100644 crypto/ml_dsa/Makefile
create mode 100644 crypto/ml_dsa/dilithium.h
create mode 100644 crypto/ml_dsa/dilithium_44.c
create mode 100644 crypto/ml_dsa/dilithium_44.h
create mode 100644 crypto/ml_dsa/dilithium_65.c
create mode 100644 crypto/ml_dsa/dilithium_65.h
create mode 100644 crypto/ml_dsa/dilithium_87.c
create mode 100644 crypto/ml_dsa/dilithium_87.h
create mode 100644 crypto/ml_dsa/dilithium_api.c
create mode 100644 crypto/ml_dsa/dilithium_debug.h
create mode 100644 crypto/ml_dsa/dilithium_ntt.c
create mode 100644 crypto/ml_dsa/dilithium_ntt.h
create mode 100644 crypto/ml_dsa/dilithium_pack.h
create mode 100644 crypto/ml_dsa/dilithium_poly.c
create mode 100644 crypto/ml_dsa/dilithium_poly.h
create mode 100644 crypto/ml_dsa/dilithium_poly_c.h
create mode 100644 crypto/ml_dsa/dilithium_poly_common.h
create mode 100644 crypto/ml_dsa/dilithium_polyvec.h
create mode 100644 crypto/ml_dsa/dilithium_polyvec_c.h
create mode 100644 crypto/ml_dsa/dilithium_reduce.h
create mode 100644 crypto/ml_dsa/dilithium_rounding.c
create mode 100644 crypto/ml_dsa/dilithium_rounding.h
create mode 100644 crypto/ml_dsa/dilithium_service_helpers.h
create mode 100644 crypto/ml_dsa/dilithium_sig.c
create mode 100644 crypto/ml_dsa/dilithium_signature_c.c
create mode 100644 crypto/ml_dsa/dilithium_signature_c.h
create mode 100644 crypto/ml_dsa/dilithium_signature_helper.c
create mode 100644 crypto/ml_dsa/dilithium_signature_impl.h
create mode 100644 crypto/ml_dsa/dilithium_type.h
create mode 100644 crypto/ml_dsa/dilithium_zetas.c
create mode 100644 crypto/ml_dsa/fips_mode.h
create mode 100644 crypto/ml_dsa/signature_domain_separation.c
create mode 100644 crypto/ml_dsa/signature_domain_separation.h
create mode 100644 crypto/ml_dsa/small_stack_support.h
diff --git a/crypto/Kconfig b/crypto/Kconfig
index a04595f9d0ca..b027460d54b7 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -1451,5 +1451,6 @@ source "drivers/crypto/Kconfig"
source "crypto/asymmetric_keys/Kconfig"
source "certs/Kconfig"
source "crypto/krb5/Kconfig"
+source "crypto/ml_dsa/Kconfig"
endif # if CRYPTO
diff --git a/crypto/Makefile b/crypto/Makefile
index e430e6e99b6a..ecd1ef79c28c 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -210,3 +210,4 @@ obj-$(CONFIG_CRYPTO_SIMD) += crypto_simd.o
obj-$(CONFIG_CRYPTO_KDF800108_CTR) += kdf_sp800108.o
obj-$(CONFIG_CRYPTO_KRB5) += krb5/
+obj-$(CONFIG_CRYPTO_ML_DSA) += ml_dsa/
diff --git a/crypto/ml_dsa/Kconfig b/crypto/ml_dsa/Kconfig
new file mode 100644
index 000000000000..cc0567c5b990
--- /dev/null
+++ b/crypto/ml_dsa/Kconfig
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: GPL-2.0
+menuconfig CRYPTO_ML_DSA
+ tristate "ML-DSA/Dilithium algorithm"
+ select CRYPTO_LIB_SHA3
+
+if CRYPTO_ML_DSA
+
+config CRYPTO_DILITHIUM_44
+ bool "Enable Dilithium-44 support"
+
+config CRYPTO_DILITHIUM_65
+ bool "Enable Dilithium-65 support"
+
+config CRYPTO_DILITHIUM_87
+ bool "Enable Dilithium-87 support"
+
+endif
diff --git a/crypto/ml_dsa/Makefile b/crypto/ml_dsa/Makefile
new file mode 100644
index 000000000000..d420a8ba6033
--- /dev/null
+++ b/crypto/ml_dsa/Makefile
@@ -0,0 +1,18 @@
+################################################################################
+# Signature implementation: Dilithium
+
+################################################################################
+# C Implementation
+################################################################################
+ml_dsa-y += \
+ signature_domain_separation.o \
+ dilithium_api.o \
+ dilithium_zetas.o \
+ dilithium_sig.o \
+ dilithium_signature_helper.o
+
+ml_dsa-$(CONFIG_CRYPTO_DILITHIUM_87) += dilithium_87.o
+ml_dsa-$(CONFIG_CRYPTO_DILITHIUM_65) += dilithium_65.o
+ml_dsa-$(CONFIG_CRYPTO_DILITHIUM_44) += dilithium_44.o
+
+obj-$(CONFIG_CRYPTO_ML_DSA) += ml_dsa.o
diff --git a/crypto/ml_dsa/dilithium.h b/crypto/ml_dsa/dilithium.h
new file mode 100644
index 000000000000..6d4982164201
--- /dev/null
+++ b/crypto/ml_dsa/dilithium.h
@@ -0,0 +1,750 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#ifndef DILITHIUM_H
+#define DILITHIUM_H
+
+#undef pr_fmt
+#define pr_fmt(fmt) "ML-DSA: " fmt
+#include <linux/types.h>
+#include <linux/errno.h>
+#include <linux/string.h>
+#include <crypto/sha2.h>
+#include <crypto/sha3.h>
+#include <crypto/hash.h>
+#include <crypto/rng.h>
+
+#define DILITHIUM_SEEDBYTES 32
+#define DILITHIUM_CRHBYTES 64
+#define DILITHIUM_TRBYTES 64
+#define DILITHIUM_RNDBYTES 32
+#define DILITHIUM_N 256
+#define DILITHIUM_Q 8380417
+#define DILITHIUM_D 13
+#define DILITHIUM_ROOT_OF_UNITY 1753
+
+extern const int32_t dilithium_zetas[DILITHIUM_N];
+#define lc_seeded_rng crypto_default_rng
+
+struct dilithium_ctx {
+ /**
+ * @brief Hash context used internally to the library - it should not
+ * be touched by the user
+ */
+ struct shake256_ctx dilithium_hash_ctx;
+
+ /**
+ * @brief When using HashML-DSA, set the hash reference used for the
+ * hash operation. Allowed values are lc_sha256, lc_sha512, lc_sha3_256,
+ * lc_sha3_384, lc_sha3_512, lc_shake128 and lc_shake256. Note, the
+ * actual message digest operation can be performed external to
+ * leancrypto. This parameter only shall indicate the used hash
+ * operation.
+ *
+ * \note Use \p dilithium_ctx_hash or
+ * \p dilithium_ed25519_ctx_hash to set this value.
+ */
+ struct crypto_shash *dilithium_prehash_type;
+
+ /**
+ * @brief length of the user context (allowed range between 0 and 255
+ * bytes)
+ *
+ * \note Use \p dilithium_ctx_userctx or
+ * \p dilithium_ed25519_ctx_userctx to set this value.
+ */
+ size_t userctxlen;
+
+ /**
+ * @brief buffer with a caller-specified context string
+ *
+ * \note Use \p dilithium_ctx_userctx or
+ * \p dilithium_ed25519_ctx_userctx to set this value.
+ */
+ const uint8_t *userctx;
+
+ /**
+ * @brief Pointer to the AHat buffer. This can be provided by the caller
+ * or it must be NULL otherwise.
+ *
+ * \note Use \p DILITHIUM_CTX_ON_STACK_AHAT to provide memory for
+ * storing AHat in the caller context and thus make the signature
+ * operation much faster starting with the 2nd use of the key (pair).
+ */
+ void *ahat;
+ unsigned short ahat_size;
+
+ /**
+ * @brief Pointer to the external mu.
+ *
+ * If set, the signature operation will use the provided mu instead of
+ * the message. In this case, the message pointer to the signature
+ * generation or verification can be NULL.
+ */
+ const uint8_t *external_mu;
+ size_t external_mu_len;
+
+ /**
+ * @brief Pointer to the randomizer
+ *
+ * This is used for the Composite signature: For the discussion of the
+ * randomizer, see https://lamps-wg.github.io/draft-composite-sigs/draft-ietf-lamps-pq-composite-sigs.html
+ */
+ const uint8_t *randomizer;
+ size_t randomizerlen;
+
+ /**
+ * @brief NIST category required for composite signatures
+ *
+ * The domain separation logic depends on the selection of the right
+ * OID for the "Domain" data.
+ */
+ unsigned int nist_category;
+
+ /**
+ * @brief When set to true, only the ML-DSA.Sign_internal or
+ * ML-DSA.Verify_internal are performed (see FIPS 204 chapter 6).
+ * Otherwise the ML-DSA.Sign / ML-DSA.Verify (see FIPS chapter 5) is
+ * applied.
+ *
+ * \note Use \p dilithium_ctx_internal or
+ * \p dilithium_ed25519_ctx_internal to set this value.
+ *
+ * \warning Only set this value to true if you exactly know what you are
+ * doing!.
+ */
+ bool ml_dsa_internal:1;
+
+ /**
+ * @brief Was aHat already filled? This is used and set internally.
+ */
+ bool ahat_expanded:1;
+} __aligned(CRYPTO_MINALIGN);
+
+static inline void dilithium_hash_init(struct dilithium_ctx *ctx)
+{
+ shake256_init(&ctx->dilithium_hash_ctx);
+}
+
+static inline bool dilithium_hash_check_blocksize(const struct dilithium_ctx *ctx, size_t bsize)
+{
+ return bsize == SHAKE256_BLOCK_SIZE;
+ //return crypto_shash_blocksize(hash_ctx->tfm) == bsize;
+}
+
+static inline bool dilithium_hash_check_digestsize(const struct dilithium_ctx *ctx, size_t dsize)
+{
+ return true;
+ //return crypto_shash_digestsize(hash_ctx->tfm) == dsize;
+}
+
+static inline void dilithium_hash_clear(struct dilithium_ctx *ctx)
+{
+ shake256_clear(&ctx->dilithium_hash_ctx);
+}
+
+static inline void dilithium_hash_update(struct dilithium_ctx *ctx,
+ const u8 *in, size_t in_size)
+{
+ shake256_update(&ctx->dilithium_hash_ctx, in, in_size);
+}
+
+static inline void dilithium_hash_finup(struct dilithium_ctx *ctx,
+ const u8 *in, size_t in_size,
+ u8 *out, size_t out_size)
+{
+ shake256_update(&ctx->dilithium_hash_ctx, in, in_size);
+ shake256_squeeze(&ctx->dilithium_hash_ctx, out, out_size);
+ shake256_clear(&ctx->dilithium_hash_ctx);
+}
+
+static inline void dilithium_hash_final(struct dilithium_ctx *ctx, u8 *out, size_t out_size)
+{
+ shake256_squeeze(&ctx->dilithium_hash_ctx, out, out_size);
+ shake256_clear(&ctx->dilithium_hash_ctx);
+}
+
+#include "dilithium_87.h"
+#include "dilithium_65.h"
+#include "dilithium_44.h"
+
+enum dilithium_type {
+ DILITHIUM_UNKNOWN, /** Unknown key type */
+ DILITHIUM_87, /** Dilithium 87 */
+ DILITHIUM_65, /** Dilithium 65 */
+ DILITHIUM_44, /** Dilithium 44 */
+};
+
+/** @defgroup Dilithium ML-DSA / CRYSTALS-Dilithium Signature Mechanism
+ *
+ * \note Although the API uses the term "dilithium", the implementation complies
+ * with FIPS 204. Thus the terms Dilithium and ML-DSA are used interchangeably.
+ *
+ * Dilithium API concept
+ *
+ * The Dilithium API is accessible via the following header files with the
+ * mentioned purpose.
+ *
+ * * dilithium.h: This API is the generic API allowing the caller to select
+ * which Dilithium type (Dilithium 87, 65 or 44) are to be used. The selection
+ * is made either with the flag specified during key generation or by matching
+ * the size of the imported data with the different dilithium_*_load API
+ * calls. All remaining APIs take the information about the Dilithium type
+ * from the provided input data.
+ *
+ * This header file only provides inline functions which selectively call
+ * the API provided with the header files below.
+ *
+ * * dilithium_87.h: Direct access to Dilithium 87.
+ *
+ * * dilithium_65.h: Direct access to Dilithium 65.
+ *
+ * * dilithium_44.h: Direct access to Dilithium 44.
+ *
+ * To support the stream mode of the Dilithium signature operation, a
+ * context structure is required. This context structure can be allocated either
+ * on the stack or heap with \p DILITHIUM_CTX_ON_STACK or
+ * \p dilithium_ctx_alloc. The context should be zeroized
+ * and freed (only for heap) with \p dilithium_ctx_zero or
+ * \p dilithium_ctx_zero_free.
+ */
+
+/**
+ * @brief Dilithium secret key
+ */
+struct dilithium_sk {
+ enum dilithium_type dilithium_type;
+ union {
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ struct dilithium_87_sk sk_87;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ struct dilithium_65_sk sk_65;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ struct dilithium_44_sk sk_44;
+#endif
+ } key;
+};
+
+/**
+ * @brief Dilithium public key
+ */
+struct dilithium_pk {
+ enum dilithium_type dilithium_type;
+ union {
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ struct dilithium_87_pk pk_87;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ struct dilithium_65_pk pk_65;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ struct dilithium_44_pk pk_44;
+#endif
+ } key;
+};
+
+/**
+ * @brief Dilithium signature
+ */
+struct dilithium_sig {
+ enum dilithium_type dilithium_type;
+ union {
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ struct dilithium_87_sig sig_87;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ struct dilithium_65_sig sig_65;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ struct dilithium_44_sig sig_44;
+#endif
+ } sig;
+};
+
+/**
+ * @ingroup Dilithium
+ * @brief Allocates Dilithium context on heap
+ *
+ * @param [out] ctx Dilithium context pointer
+ *
+ * @return 0 (success) or < 0 on error
+ */
+struct dilithium_ctx *dilithium_ctx_alloc(void);
+
+/**
+ * @ingroup Dilithium
+ * @brief Allocates Dilithium context on heap with support to keep the internal
+ * representation of the key.
+ *
+ * \note See \p DILITHIUM_CTX_ON_STACK_AHAT for details.
+ *
+ * @param [out] ctx Dilithium context pointer
+ *
+ * @return 0 (success) or < 0 on error
+ */
+struct dilithium_ctx *dilithium_ctx_alloc_ahat(enum dilithium_type type);
+
+/**
+ * @ingroup Dilithium
+ * @brief Zeroizes and frees Dilithium context on heap
+ *
+ * @param [out] ctx Dilithium context pointer
+ */
+void dilithium_ctx_zero_free(struct dilithium_ctx *ctx);
+
+/**
+ * @ingroup Dilithium
+ * @brief Zeroizes Dilithium context either on heap or on stack
+ *
+ * @param [out] ctx Dilithium context pointer
+ */
+void dilithium_ctx_zero(struct dilithium_ctx *ctx);
+
+/**
+ * @ingroup Dilithium
+ * @brief Mark the Dilithium context to execute ML-DSA.Sign_internal /
+ * ML-DSA.Verify_internal.
+ *
+ * @param [in] ctx Dilithium context
+ */
+void dilithium_ctx_internal(struct dilithium_ctx *ctx);
+
+/**
+ * @ingroup Dilithium
+ * @brief Set the hash type that was used for pre-hashing the message. The
+ * message digest is used with the HashML-DSA. The message digest
+ * is to be provided via the message pointer in the sign/verify APIs.
+ *
+ * @param [in] ctx Dilithium context
+ * @param [in] hash Hash context referencing the used hash for pre-hashing the
+ * message
+ */
+void dilithium_ctx_hash(struct dilithium_ctx *ctx,
+ struct crypto_shash *hash);
+
+/**
+ * @ingroup Dilithium
+ * @brief Specify the optional user context string to be applied with the
+ * Dilithium signature operation.
+ *
+ * @param [in] ctx Dilithium context
+ * @param [in] userctx User context string
+ * @param [in] userctxlen Size of the user context string
+ */
+void dilithium_ctx_userctx(struct dilithium_ctx *ctx,
+ const uint8_t *userctx, size_t userctxlen);
+
+/**
+ * @ingroup Dilithium
+ * @brief Specify the optional external mu value.
+ *
+ * \note If the external mu is specified, the signature generation /
+ * verification APIs do not require a message. In this case, the message buffer
+ * can be set to NULL.
+ *
+ * \note If both a message and an external mu are provided, the external mu
+ * takes precedence.
+ *
+ * @param [in] ctx Dilithium context
+ * @param [in] external_mu User context string
+ * @param [in] external_mu_len Size of the user context string
+ */
+void dilithium_ctx_external_mu(struct dilithium_ctx *ctx,
+ const uint8_t *external_mu,
+ size_t external_mu_len);
+
+/**
+ * @ingroup Dilithium
+ * @brief Invalidate the expanded key that potentially is stored in the context.
+ *
+ * This call can be executed on a context irrespective it was allocated with
+ * space for the expanded representation or not. Thus, the caller does not need
+ * to track whether the context supports the expanded key.
+ *
+ * @param [in] ctx Dilithium context
+ */
+void dilithium_ctx_drop_ahat(struct dilithium_ctx *ctx);
+
+/**
+ * @ingroup Dilithium
+ * @brief Obtain Dilithium type from secret key
+ *
+ * @param [in] sk Secret key from which the type is to be obtained
+ *
+ * @return key type
+ */
+enum dilithium_type dilithium_sk_type(const struct dilithium_sk *sk);
+
+/**
+ * @ingroup Dilithium
+ * @brief Obtain Dilithium type from public key
+ *
+ * @param [in] pk Public key from which the type is to be obtained
+ *
+ * @return key type
+ */
+enum dilithium_type dilithium_pk_type(const struct dilithium_pk *pk);
+
+/**
+ * @ingroup Dilithium
+ * @brief Obtain Dilithium type from signature
+ *
+ * @param [in] sig Signature from which the type is to be obtained
+ *
+ * @return key type
+ */
+enum dilithium_type
+dilithium_sig_type(const struct dilithium_sig *sig);
+
+/**
+ * @ingroup Dilithium
+ * @brief Return the size of the Dilithium secret key.
+ *
+ * @param [in] dilithium_type Dilithium type for which the size is requested
+ *
+ * @return requested size
+ */
+__pure unsigned int
+dilithium_sk_size(enum dilithium_type dilithium_type);
+
+/**
+ * @ingroup Dilithium
+ * @brief Return the size of the Dilithium public key.
+ *
+ * @param [in] dilithium_type Dilithium type for which the size is requested
+ *
+ * @return requested size
+ */
+__pure unsigned int
+dilithium_pk_size(enum dilithium_type dilithium_type);
+
+/**
+ * @ingroup Dilithium
+ * @brief Return the size of the Dilithium signature.
+ *
+ * @param [in] dilithium_type Dilithium type for which the size is requested
+ *
+ * @return requested size
+ */
+unsigned int __pure
+dilithium_sig_size(enum dilithium_type dilithium_type);
+
+/**
+ * @ingroup Dilithium
+ * @brief Load a Dilithium secret key provided with a buffer into the leancrypto
+ * data structure.
+ *
+ * @param [out] sk Secret key to be filled (the caller must have it allocated)
+ * @param [in] src_key Buffer that holds the key to be imported
+ * @param [in] src_key_len Buffer length that holds the key to be imported
+ *
+ * @return 0 on success or < 0 on error
+ */
+int dilithium_sk_load(struct dilithium_sk *sk, const uint8_t *src_key,
+ size_t src_key_len);
+
+/**
+ * @ingroup Dilithium
+ * @brief Load a Dilithium public key provided with a buffer into the leancrypto
+ * data structure.
+ *
+ * @param [out] pk Secret key to be filled (the caller must have it allocated)
+ * @param [in] src_key Buffer that holds the key to be imported
+ * @param [in] src_key_len Buffer length that holds the key to be imported
+ *
+ * @return 0 on success or < 0 on error
+ */
+int dilithium_pk_load(struct dilithium_pk *pk, const uint8_t *src_key,
+ size_t src_key_len);
+
+/**
+ * @ingroup Dilithium
+ * @brief Load a Dilithium signature provided with a buffer into the leancrypto
+ * data structure.
+ *
+ * @param [out] sig Secret key to be filled (the caller must have it allocated)
+ * @param [in] src_sig Buffer that holds the signature to be imported
+ * @param [in] src_sig_len Buffer length that holds the signature to be imported
+ *
+ * @return 0 on success or < 0 on error
+ */
+int dilithium_sig_load(struct dilithium_sig *sig, const uint8_t *src_sig,
+ size_t src_sig_len);
+
+/**
+ * @ingroup Dilithium
+ * @brief Obtain the reference to the Dilithium key and its length
+ *
+ * \note Only pointer references into the leancrypto data structure are returned
+ * which implies that any modification will modify the leancrypto key, too.
+ *
+ * @param [out] dilithium_key Dilithium key pointer
+ * @param [out] dilithium_key_len Length of the key buffer
+ * @param [in] sk Dilithium secret key from which the references are obtained
+ *
+ * @return 0 on success, != 0 on error
+ */
+int dilithium_sk_ptr(uint8_t **dilithium_key, size_t *dilithium_key_len,
+ struct dilithium_sk *sk);
+
+/**
+ * @ingroup Dilithium
+ * @brief Obtain the reference to the Dilithium key and its length
+ *
+ * \note Only pointer references into the leancrypto data structure are returned
+ * which implies that any modification will modify the leancrypto key, too.
+ *
+ * @param [out] dilithium_key Dilithium key pointer
+ * @param [out] dilithium_key_len Length of the key buffer
+ * @param [in] pk Dilithium publi key from which the references are obtained
+ *
+ * @return 0 on success, != 0 on error
+ */
+int dilithium_pk_ptr(uint8_t **dilithium_key, size_t *dilithium_key_len,
+ struct dilithium_pk *pk);
+
+/**
+ * @ingroup Dilithium
+ * @brief Obtain the reference to the Dilithium signature and its length
+ *
+ * \note Only pointer references into the leancrypto data structure are returned
+ * which implies that any modification will modify the leancrypto signature,
+ * too.
+ *
+ * @param [out] dilithium_sig Dilithium signature pointer
+ * @param [out] dilithium_sig_len Length of the signature buffer
+ * @param [in] sig Dilithium signature from which the references are obtained
+ *
+ * @return 0 on success, != 0 on error
+ */
+int dilithium_sig_ptr(uint8_t **dilithium_sig, size_t *dilithium_sig_len,
+ struct dilithium_sig *sig);
+
+/**
+ * @ingroup Dilithium
+ * @brief Computes signature in one shot
+ *
+ * @param [out] sig pointer to output signature
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ * @param [in] sk pointer to bit-packed secret key
+ * @param [in] rng_ctx pointer to seeded random number generator context - when
+ * pointer is non-NULL, perform a randomized signing.
+ * Otherwise use deterministic signing.
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_sign(struct dilithium_sig *sig, const uint8_t *m,
+ size_t mlen, const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx);
+
+/**
+ * @ingroup Dilithium
+ * @brief Computes signature woth user context in one shot
+ *
+ * This API allows the caller to provide an arbitrary context buffer which
+ * is hashed together with the message to form the message digest to be signed.
+ *
+ * Using the ctx structure, the caller can select 3 different types of ML-DSA:
+ *
+ * * ctx->dilithium_prehash_type set to a hash type, HashML-DSA is assumed which
+ * implies that the message m must be exactly digest size (FIPS 204 section
+ * 5.4)
+ *
+ * * ctx->ml_dsa_internal set to 1, the ML-DSA.Sign_internal and
+ * .Verify_internal are executed (FIPS 204 chapter 6)
+ *
+ * * both aforementioned parameter set to NULL / 0, ML-DSA.Sign and
+ * ML-DSA.Verify are executed (FIPS 204 sections 5.2 and 5.3)
+ *
+ * @param [out] sig pointer to output signature
+ * @param [in] ctx reference to the allocated Dilithium context handle
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ * @param [in] sk pointer to bit-packed secret key
+ * @param [in] rng_ctx pointer to seeded random number generator context - when
+ * pointer is non-NULL, perform a randomized signing.
+ * Otherwise use deterministic signing.
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_sign_ctx(struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen, const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx);
+
+/**
+ * @ingroup Dilithium
+ * @brief Initializes a signature operation
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_sign_update and dilithium_sign_final.
+ *
+ * @param [in,out] ctx pointer Dilithium context
+ * @param [in] sk pointer to bit-packed secret key
+ *
+ * @return 0 (success) or < 0 on error; -EOPNOTSUPP is returned if a different
+ * hash than lc_shake256 is used.
+ */
+int dilithium_sign_init(struct dilithium_ctx *ctx,
+ const struct dilithium_sk *sk);
+
+/**
+ * @ingroup Dilithium
+ * @brief Add more data to an already initialized signature state
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_sign_init and dilithium_sign_final.
+ *
+ * @param [in] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_sign_update(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen);
+
+/**
+ * @ingroup Dilithium
+ * @brief Computes signature
+ *
+ * @param [out] sig pointer to output signature
+ * @param [in] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init and filled with
+ * dilithium_sign_update
+ * @param [in] sk pointer to bit-packed secret key
+ * @param [in] rng_ctx pointer to seeded random number generator context - when
+ * pointer is non-NULL, perform a randomized signing.
+ * Otherwise use deterministic signing.
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_sign_final(struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx);
+
+/**
+ * @ingroup Dilithium
+ * @brief Verifies signature in one shot
+ *
+ * @param [in] sig pointer to input signature
+ * @param [in] m pointer to message
+ * @param [in] mlen length of message
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 if signature could be verified correctly and -EBADMSG when
+ * signature cannot be verified, < 0 on other errors
+ */
+int dilithium_verify(const struct dilithium_sig *sig, const uint8_t *m,
+ size_t mlen, const struct dilithium_pk *pk);
+
+/**
+ * @ingroup Dilithium
+ * @brief Verifies signature with Dilithium context in one shot
+ *
+ * This API allows the caller to provide an arbitrary context buffer which
+ * is hashed together with the message to form the message digest to be signed.
+ *
+ * @param [in] sig pointer to input signature
+ * @param [in] ctx reference to the allocated Dilithium context handle
+ * @param [in] m pointer to message
+ * @param [in] mlen length of message
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 if signature could be verified correctly and -EBADMSG when
+ * signature cannot be verified, < 0 on other errors
+ */
+int dilithium_verify_ctx(const struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen, const struct dilithium_pk *pk);
+
+/**
+ * @ingroup Dilithium
+ * @brief Initializes a signature verification operation
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_verify_update and
+ * dilithium_verify_final.
+ *
+ * @param [in,out] ctx pointer to an allocated Dilithium context
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 (success) or < 0 on error; -EOPNOTSUPP is returned if a different
+ * hash than lc_shake256 is used.
+ */
+int dilithium_verify_init(struct dilithium_ctx *ctx,
+ const struct dilithium_pk *pk);
+
+/**
+ * @ingroup Dilithium
+ * @brief Add more data to an already initialized signature state
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_verify_init and
+ * dilithium_verify_final.
+ *
+ * @param [in] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_verify_update(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen);
+
+/**
+ * @ingroup Dilithium
+ * @brief Verifies signature
+ *
+ * @param [in] sig pointer to output signature
+ * @param [in] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init and filled with
+ * dilithium_sign_update
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 if signature could be verified correctly and -EBADMSG when
+ * signature cannot be verified, < 0 on other errors
+ */
+int dilithium_verify_final(const struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_pk *pk);
+
+/* No valgrind */
+#define poison(addr, len)
+#define unpoison(addr, len)
+#define is_poisoned(addr, len)
+
+#endif /* DILITHIUM_H */
diff --git a/crypto/ml_dsa/dilithium_44.c b/crypto/ml_dsa/dilithium_44.c
new file mode 100644
index 000000000000..1aea716016f0
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_44.c
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+#define DILITHIUM_TYPE_44
+#include "dilithium.h"
+
+#define DILITHIUM_TYPE "44"
+#define DILITHIUM_MODE 2
+#define DILITHIUM_NIST_CATEGORY DILITHIUM44_NIST_CATEGORY
+#define DILITHIUM_LAMBDA DILITHIUM44_LAMBDA
+#define DILITHIUM_K DILITHIUM44_K
+#define DILITHIUM_L DILITHIUM44_L
+#define DILITHIUM_ETA DILITHIUM44_ETA
+#define DILITHIUM_TAU DILITHIUM44_TAU
+#define DILITHIUM_BETA DILITHIUM44_BETA
+#define DILITHIUM_GAMMA1 DILITHIUM44_GAMMA1
+#define DILITHIUM_GAMMA2 DILITHIUM44_GAMMA2
+#define DILITHIUM_OMEGA DILITHIUM44_OMEGA
+
+#define DILITHIUM_CTILDE_BYTES DILITHIUM44_CTILDE_BYTES
+#define DILITHIUM_POLYT1_PACKEDBYTES DILITHIUM44_POLYT1_PACKEDBYTES
+#define DILITHIUM_POLYT0_PACKEDBYTES DILITHIUM44_POLYT0_PACKEDBYTES
+#define DILITHIUM_POLYVECH_PACKEDBYTES DILITHIUM44_POLYVECH_PACKEDBYTES
+#define DILITHIUM_POLYZ_PACKEDBYTES DILITHIUM44_POLYZ_PACKEDBYTES
+#define DILITHIUM_POLYW1_PACKEDBYTES DILITHIUM44_POLYW1_PACKEDBYTES
+#define DILITHIUM_POLYETA_PACKEDBYTES DILITHIUM44_POLYETA_PACKEDBYTES
+
+#define DILITHIUM_PUBLICKEYBYTES DILITHIUM44_PUBLICKEYBYTES
+#define DILITHIUM_SECRETKEYBYTES DILITHIUM44_SECRETKEYBYTES
+#define DILITHIUM_CRYPTO_BYTES DILITHIUM44_CRYPTO_BYTES
+
+#include "dilithium_ntt.c"
+#include "dilithium_poly.c"
+#include "dilithium_rounding.c"
+#include "dilithium_signature_c.c"
diff --git a/crypto/ml_dsa/dilithium_44.h b/crypto/ml_dsa/dilithium_44.h
new file mode 100644
index 000000000000..6490c83e7100
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_44.h
@@ -0,0 +1,377 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#ifndef DILITHIUM_44_H
+#define DILITHIUM_44_H
+
+/*
+ * Dilithium Security Levels
+ * 2 -> 192 bits of security strength
+ * 3 -> 225 bits of security strength
+ * 5 -> 257 bits of security strength
+ */
+
+/* DILITHIUM_MODE 2 */
+#define DILITHIUM44_NIST_CATEGORY 1
+#define DILITHIUM44_LAMBDA 128
+#define DILITHIUM44_K 4
+#define DILITHIUM44_L 4
+#define DILITHIUM44_ETA 2
+#define DILITHIUM44_TAU 39
+#define DILITHIUM44_BETA 78
+#define DILITHIUM44_GAMMA1 (1 << 17)
+#define DILITHIUM44_GAMMA2 ((DILITHIUM_Q - 1) / 88)
+#define DILITHIUM44_OMEGA 80
+
+#define DILITHIUM44_CTILDE_BYTES (DILITHIUM44_LAMBDA * 2 / 8)
+#define DILITHIUM44_POLYT1_PACKEDBYTES 320
+#define DILITHIUM44_POLYT0_PACKEDBYTES 416
+#define DILITHIUM44_POLYVECH_PACKEDBYTES (DILITHIUM44_OMEGA + DILITHIUM44_K)
+
+#if DILITHIUM44_GAMMA1 == (1 << 17)
+#define DILITHIUM44_POLYZ_PACKEDBYTES 576
+#elif DILITHIUM44_GAMMA1 == (1 << 19)
+#define DILITHIUM44_POLYZ_PACKEDBYTES 640
+#endif
+
+#if DILITHIUM44_GAMMA2 == (DILITHIUM_Q - 1) / 88
+#define DILITHIUM44_POLYW1_PACKEDBYTES 192
+#elif DILITHIUM44_GAMMA2 == (DILITHIUM_Q - 1) / 32
+#define DILITHIUM44_POLYW1_PACKEDBYTES 128
+#endif
+
+#if DILITHIUM44_ETA == 2
+#define DILITHIUM44_POLYETA_PACKEDBYTES 96
+#elif DILITHIUM44_ETA == 4
+#define DILITHIUM44_POLYETA_PACKEDBYTES 128
+#endif
+
+/*
+ * Sizes of the different Dilithium buffer types.
+ *
+ * WARNING: Do not use these defines in your code. If you need the sizes of
+ * the different variable sizes, use sizeof of the different variable structs or
+ * use the different *_size functions documented below to retrieve the data size
+ * of a particular Dilithium component.
+ */
+#define DILITHIUM44_PUBLICKEYBYTES \
+ (DILITHIUM_SEEDBYTES + \
+ DILITHIUM44_K * DILITHIUM44_POLYT1_PACKEDBYTES)
+#define DILITHIUM44_SECRETKEYBYTES \
+ (2 * DILITHIUM_SEEDBYTES + DILITHIUM_TRBYTES + \
+ DILITHIUM44_L * DILITHIUM44_POLYETA_PACKEDBYTES + \
+ DILITHIUM44_K * DILITHIUM44_POLYETA_PACKEDBYTES + \
+ DILITHIUM44_K * DILITHIUM44_POLYT0_PACKEDBYTES)
+
+#define DILITHIUM44_CRYPTO_BYTES \
+ (DILITHIUM44_CTILDE_BYTES + \
+ DILITHIUM44_L * DILITHIUM44_POLYZ_PACKEDBYTES + \
+ DILITHIUM44_POLYVECH_PACKEDBYTES)
+
+#ifndef __ASSEMBLER__
+/**
+ * @brief Dilithium secret key
+ */
+struct dilithium_44_sk {
+ uint8_t sk[DILITHIUM44_SECRETKEYBYTES];
+};
+
+/**
+ * @brief Dilithium public key
+ */
+struct dilithium_44_pk {
+ uint8_t pk[DILITHIUM44_PUBLICKEYBYTES];
+};
+
+/**
+ * @brief Dilithium signature
+ */
+struct dilithium_44_sig {
+ uint8_t sig[DILITHIUM44_CRYPTO_BYTES];
+};
+
+/*
+ * The alignment is based on largest alignment of a polyvecl typedef - this is
+ * the AVX2 definition.
+ */
+#define DILITHIUM_AHAT_ALIGNMENT (32)
+
+/* Size of the AHat matrix for ML-DSA 87 */
+#define DILITHIUM_44_AHAT_SIZE \
+ (256 * sizeof(int32_t) * DILITHIUM44_K * DILITHIUM44_L)
+
+/**
+ * @brief Zeroize Dilithium context allocated with
+ * DILITHIUM_CTX_ON_STACK dilithium_ed25519_alloc
+ *
+ * @param [in] ctx Dilithium context to be zeroized
+ */
+static inline void dilithium_44_ctx_zero(struct dilithium_ctx *ctx)
+{
+ if (!ctx)
+ return;
+ dilithium_hash_clear(ctx);
+ if (ctx->ahat) {
+ memzero_explicit(ctx->ahat, ctx->ahat_size);
+ ctx->ahat_expanded = 0;
+ }
+}
+
+/**
+ * @brief Allocate Dilithium stream context on heap
+ *
+ * @param [out] ctx Allocated Dilithium stream context
+ *
+ * @return: 0 on success, < 0 on error
+ */
+struct dilithium_ctx *dilithium_44_ctx_alloc(void);
+
+/**
+ * @brief Allocate Dilithium stream context on heap including additional
+ * parameter relevant for the signature operation.
+ *
+ * \note See \p DILITHIUM_44_CTX_ON_STACK_AHAT for details.
+ *
+ * @param [out] ctx Allocated Dilithium stream context
+ *
+ * @return: 0 on success, < 0 on error
+ */
+struct dilithium_ctx *dilithium_44_ctx_alloc_ahat(void);
+
+/**
+ * @brief Zeroize and free Dilithium stream context
+ *
+ * @param [in] ctx Dilithium stream context to be zeroized and freed
+ */
+void dilithium_44_ctx_zero_free(struct dilithium_ctx *ctx);
+
+/**
+ * @brief Return the size of the Dilithium secret key.
+ */
+__pure
+static inline unsigned int dilithium_44_sk_size(void)
+{
+ return sizeof_field(struct dilithium_44_sk, sk);
+}
+
+/**
+ * @brief Return the size of the Dilithium public key.
+ */
+__pure
+static inline unsigned int dilithium_44_pk_size(void)
+{
+ return sizeof_field(struct dilithium_44_pk, pk);
+}
+
+/**
+ * @brief Return the size of the Dilithium signature.
+ */
+__pure
+static inline unsigned int dilithium_44_sig_size(void)
+{
+ return sizeof_field(struct dilithium_44_sig, sig);
+}
+
+/**
+ * @brief Computes ML-DSA signature in one shot
+ *
+ * @param [out] sig pointer to output signature
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ * @param [in] sk pointer to bit-packed secret key
+ * @param [in] rng_ctx pointer to seeded random number generator context - when
+ * pointer is non-NULL, perform a randomized signing.
+ * Otherwise use deterministic signing.
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_44_sign(struct dilithium_44_sig *sig, const uint8_t *m,
+ size_t mlen, const struct dilithium_44_sk *sk,
+ struct crypto_rng *rng_ctx);
+
+/**
+ * @brief Computes signature with Dilithium context in one shot
+ *
+ * This API allows the caller to provide an arbitrary context buffer which
+ * is hashed together with the message to form the message digest to be signed.
+ *
+ * @param [out] sig pointer to output signature
+ * @param [in] ctx reference to the allocated Dilithium context handle
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ * @param [in] sk pointer to bit-packed secret key
+ * @param [in] rng_ctx pointer to seeded random number generator context - when
+ * pointer is non-NULL, perform a randomized signing.
+ * Otherwise use deterministic signing.
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_44_sign_ctx(struct dilithium_44_sig *sig,
+ struct dilithium_ctx *ctx,
+ const uint8_t *m, size_t mlen,
+ const struct dilithium_44_sk *sk,
+ struct crypto_rng *rng_ctx);
+
+/**
+ * @brief Initializes a signature operation
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_sign_update and dilithium_sign_final.
+ *
+ * @param [in,out] ctx pointer to an allocated Dilithium context
+ * @param [in] sk pointer to bit-packed secret key
+ *
+ * @return 0 (success) or < 0 on error; -EOPNOTSUPP is returned if a different
+ * hash than lc_shake256 is used.
+ */
+int dilithium_44_sign_init(struct dilithium_ctx *ctx,
+ const struct dilithium_44_sk *sk);
+
+/**
+ * @brief Add more data to an already initialized signature state
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_sign_init and dilithium_sign_final.
+ *
+ * @param [in] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_44_sign_update(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen);
+
+/**
+ * @brief Computes signature
+ *
+ * @param [out] sig pointer to output signature
+ * @param [in] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init and filled with
+ * dilithium_sign_update
+ * @param [in] sk pointer to bit-packed secret key
+ * @param [in] rng_ctx pointer to seeded random number generator context - when
+ * pointer is non-NULL, perform a randomized signing.
+ * Otherwise use deterministic signing.
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_44_sign_final(struct dilithium_44_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_44_sk *sk,
+ struct crypto_rng *rng_ctx);
+
+/**
+ * @brief Verifies ML-DSA signature in one shot
+ *
+ * @param [in] sig pointer to input signature
+ * @param [in] m pointer to message
+ * @param [in] mlen length of message
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 if signature could be verified correctly and -EBADMSG when
+ * signature cannot be verified, < 0 on other errors
+ */
+int dilithium_44_verify(const struct dilithium_44_sig *sig, const uint8_t *m,
+ size_t mlen, const struct dilithium_44_pk *pk);
+
+/**
+ * @brief Verifies signature with Dilithium context in one shot
+ *
+ * This API allows the caller to provide an arbitrary context buffer which
+ * is hashed together with the message to form the message digest to be signed.
+ *
+ * @param [in] sig pointer to input signature
+ * @param [in] ctx reference to the allocated Dilithium context handle
+ * @param [in] m pointer to message
+ * @param [in] mlen length of message
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 if signature could be verified correctly and -EBADMSG when
+ * signature cannot be verified, < 0 on other errors
+ */
+int dilithium_44_verify_ctx(const struct dilithium_44_sig *sig,
+ struct dilithium_ctx *ctx,
+ const uint8_t *m, size_t mlen,
+ const struct dilithium_44_pk *pk);
+
+/**
+ * @brief Initializes a signature verification operation
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_verify_update and
+ * dilithium_verify_final.
+ *
+ * @param [in,out] ctx pointer to an allocated Dilithium context
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 (success) or < 0 on error; -EOPNOTSUPP is returned if a different
+ * hash than lc_shake256 is used.
+ */
+int dilithium_44_verify_init(struct dilithium_ctx *ctx,
+ const struct dilithium_44_pk *pk);
+
+/**
+ * @brief Add more data to an already initialized signature state
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_verify_init and
+ * dilithium_verify_final.
+ *
+ * @param [in,out] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_44_verify_update(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen);
+
+/**
+ * @brief Verifies signature
+ *
+ * @param [in] sig pointer to output signature
+ * @param [in] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init and filled with
+ * dilithium_sign_update
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 if signature could be verified correctly and -EBADMSG when
+ * signature cannot be verified, < 0 on other errors
+ */
+int dilithium_44_verify_final(const struct dilithium_44_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_44_pk *pk);
+
+#endif /* __ASSEMBLER__ */
+
+#endif /* DILITHIUM_44_H */
diff --git a/crypto/ml_dsa/dilithium_65.c b/crypto/ml_dsa/dilithium_65.c
new file mode 100644
index 000000000000..08f3a8e71228
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_65.c
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+#define DILITHIUM_TYPE_65
+#include "dilithium.h"
+
+#define DILITHIUM_TYPE "65"
+#define DILITHIUM_MODE 3
+#define DILITHIUM_NIST_CATEGORY DILITHIUM65_NIST_CATEGORY
+#define DILITHIUM_LAMBDA DILITHIUM65_LAMBDA
+#define DILITHIUM_K DILITHIUM65_K
+#define DILITHIUM_L DILITHIUM65_L
+#define DILITHIUM_ETA DILITHIUM65_ETA
+#define DILITHIUM_TAU DILITHIUM65_TAU
+#define DILITHIUM_BETA DILITHIUM65_BETA
+#define DILITHIUM_GAMMA1 DILITHIUM65_GAMMA1
+#define DILITHIUM_GAMMA2 DILITHIUM65_GAMMA2
+#define DILITHIUM_OMEGA DILITHIUM65_OMEGA
+
+#define DILITHIUM_CTILDE_BYTES DILITHIUM65_CTILDE_BYTES
+#define DILITHIUM_POLYT1_PACKEDBYTES DILITHIUM65_POLYT1_PACKEDBYTES
+#define DILITHIUM_POLYT0_PACKEDBYTES DILITHIUM65_POLYT0_PACKEDBYTES
+#define DILITHIUM_POLYVECH_PACKEDBYTES DILITHIUM65_POLYVECH_PACKEDBYTES
+#define DILITHIUM_POLYZ_PACKEDBYTES DILITHIUM65_POLYZ_PACKEDBYTES
+#define DILITHIUM_POLYW1_PACKEDBYTES DILITHIUM65_POLYW1_PACKEDBYTES
+#define DILITHIUM_POLYETA_PACKEDBYTES DILITHIUM65_POLYETA_PACKEDBYTES
+
+#define DILITHIUM_PUBLICKEYBYTES DILITHIUM65_PUBLICKEYBYTES
+#define DILITHIUM_SECRETKEYBYTES DILITHIUM65_SECRETKEYBYTES
+#define DILITHIUM_CRYPTO_BYTES DILITHIUM65_CRYPTO_BYTES
+
+#include "dilithium_ntt.c"
+#include "dilithium_poly.c"
+#include "dilithium_rounding.c"
+#include "dilithium_signature_c.c"
diff --git a/crypto/ml_dsa/dilithium_65.h b/crypto/ml_dsa/dilithium_65.h
new file mode 100644
index 000000000000..0a36f7eed63e
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_65.h
@@ -0,0 +1,377 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#ifndef DILITHIUM_65_H
+#define DILITHIUM_65_H
+
+/*
+ * Dilithium Security Levels
+ * 2 -> 192 bits of security strength
+ * 3 -> 225 bits of security strength
+ * 5 -> 257 bits of security strength
+ */
+
+/* DILITHIUM_MODE 3 */
+#define DILITHIUM65_NIST_CATEGORY 3
+#define DILITHIUM65_LAMBDA 192
+#define DILITHIUM65_K 6
+#define DILITHIUM65_L 5
+#define DILITHIUM65_ETA 4
+#define DILITHIUM65_TAU 49
+#define DILITHIUM65_BETA 196
+#define DILITHIUM65_GAMMA1 (1 << 19)
+#define DILITHIUM65_GAMMA2 ((DILITHIUM_Q - 1) / 32)
+#define DILITHIUM65_OMEGA 55
+
+#define DILITHIUM65_CTILDE_BYTES (DILITHIUM65_LAMBDA * 2 / 8)
+#define DILITHIUM65_POLYT1_PACKEDBYTES 320
+#define DILITHIUM65_POLYT0_PACKEDBYTES 416
+#define DILITHIUM65_POLYVECH_PACKEDBYTES (DILITHIUM65_OMEGA + DILITHIUM65_K)
+
+#if DILITHIUM65_GAMMA1 == (1 << 17)
+#define DILITHIUM65_POLYZ_PACKEDBYTES 576
+#elif DILITHIUM65_GAMMA1 == (1 << 19)
+#define DILITHIUM65_POLYZ_PACKEDBYTES 640
+#endif
+
+#if DILITHIUM65_GAMMA2 == (DILITHIUM_Q - 1) / 88
+#define DILITHIUM65_POLYW1_PACKEDBYTES 192
+#elif DILITHIUM65_GAMMA2 == (DILITHIUM_Q - 1) / 32
+#define DILITHIUM65_POLYW1_PACKEDBYTES 128
+#endif
+
+#if DILITHIUM65_ETA == 2
+#define DILITHIUM65_POLYETA_PACKEDBYTES 96
+#elif DILITHIUM65_ETA == 4
+#define DILITHIUM65_POLYETA_PACKEDBYTES 128
+#endif
+
+/*
+ * Sizes of the different Dilithium buffer types.
+ *
+ * WARNING: Do not use these defines in your code. If you need the sizes of
+ * the different variable sizes, use sizeof of the different variable structs or
+ * use the different *_size functions documented below to retrieve the data size
+ * of a particular Dilithium component.
+ */
+#define DILITHIUM65_PUBLICKEYBYTES \
+ (DILITHIUM_SEEDBYTES + \
+ DILITHIUM65_K * DILITHIUM65_POLYT1_PACKEDBYTES)
+#define DILITHIUM65_SECRETKEYBYTES \
+ (2 * DILITHIUM_SEEDBYTES + DILITHIUM_TRBYTES + \
+ DILITHIUM65_L * DILITHIUM65_POLYETA_PACKEDBYTES + \
+ DILITHIUM65_K * DILITHIUM65_POLYETA_PACKEDBYTES + \
+ DILITHIUM65_K * DILITHIUM65_POLYT0_PACKEDBYTES)
+
+#define DILITHIUM65_CRYPTO_BYTES \
+ (DILITHIUM65_CTILDE_BYTES + \
+ DILITHIUM65_L * DILITHIUM65_POLYZ_PACKEDBYTES + \
+ DILITHIUM65_POLYVECH_PACKEDBYTES)
+
+#ifndef __ASSEMBLER__
+/**
+ * @brief Dilithium secret key
+ */
+struct dilithium_65_sk {
+ uint8_t sk[DILITHIUM65_SECRETKEYBYTES];
+};
+
+/**
+ * @brief Dilithium public key
+ */
+struct dilithium_65_pk {
+ uint8_t pk[DILITHIUM65_PUBLICKEYBYTES];
+};
+
+/**
+ * @brief Dilithium signature
+ */
+struct dilithium_65_sig {
+ uint8_t sig[DILITHIUM65_CRYPTO_BYTES];
+};
+
+/*
+ * The alignment is based on largest alignment of a polyvecl typedef - this is
+ * the AVX2 definition.
+ */
+#define DILITHIUM_AHAT_ALIGNMENT (32)
+
+/* Size of the AHat matrix for ML-DSA 87 */
+#define DILITHIUM_65_AHAT_SIZE \
+ (256 * sizeof(int32_t) * DILITHIUM65_K * DILITHIUM65_L)
+
+/**
+ * @brief Zeroize Dilithium context allocated with
+ * DILITHIUM_CTX_ON_STACK dilithium_ed25519_alloc
+ *
+ * @param [in] ctx Dilithium context to be zeroized
+ */
+static inline void dilithium_65_ctx_zero(struct dilithium_ctx *ctx)
+{
+ if (!ctx)
+ return;
+ dilithium_hash_clear(ctx);
+ if (ctx->ahat) {
+ memzero_explicit(ctx->ahat, ctx->ahat_size);
+ ctx->ahat_expanded = 0;
+ }
+}
+
+/**
+ * @brief Allocate Dilithium stream context on heap
+ *
+ * @param [out] ctx Allocated Dilithium stream context
+ *
+ * @return: 0 on success, < 0 on error
+ */
+struct dilithium_ctx *dilithium_65_ctx_alloc(void);
+
+/**
+ * @brief Allocate Dilithium stream context on heap including additional
+ * parameter relevant for the signature operation.
+ *
+ * \note See \p DILITHIUM_65_CTX_ON_STACK_AHAT for details.
+ *
+ * @param [out] ctx Allocated Dilithium stream context
+ *
+ * @return: 0 on success, < 0 on error
+ */
+struct dilithium_ctx *dilithium_65_ctx_alloc_ahat(void);
+
+/**
+ * @brief Zeroize and free Dilithium stream context
+ *
+ * @param [in] ctx Dilithium stream context to be zeroized and freed
+ */
+void dilithium_65_ctx_zero_free(struct dilithium_ctx *ctx);
+
+/**
+ * @brief Return the size of the Dilithium secret key.
+ */
+__pure
+static inline unsigned int dilithium_65_sk_size(void)
+{
+ return sizeof_field(struct dilithium_65_sk, sk);
+}
+
+/**
+ * @brief Return the size of the Dilithium public key.
+ */
+__pure
+static inline unsigned int dilithium_65_pk_size(void)
+{
+ return sizeof_field(struct dilithium_65_pk, pk);
+}
+
+/**
+ * @brief Return the size of the Dilithium signature.
+ */
+__pure
+static inline unsigned int dilithium_65_sig_size(void)
+{
+ return sizeof_field(struct dilithium_65_sig, sig);
+}
+
+/**
+ * @brief Computes ML-DSA signature in one shot
+ *
+ * @param [out] sig pointer to output signature
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ * @param [in] sk pointer to bit-packed secret key
+ * @param [in] rng_ctx pointer to seeded random number generator context - when
+ * pointer is non-NULL, perform a randomized signing.
+ * Otherwise use deterministic signing.
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_65_sign(struct dilithium_65_sig *sig, const uint8_t *m,
+ size_t mlen, const struct dilithium_65_sk *sk,
+ struct crypto_rng *rng_ctx);
+
+/**
+ * @brief Computes signature with Dilithium context in one shot
+ *
+ * This API allows the caller to provide an arbitrary context buffer which
+ * is hashed together with the message to form the message digest to be signed.
+ *
+ * @param [out] sig pointer to output signature
+ * @param [in] ctx reference to the allocated Dilithium context handle
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ * @param [in] sk pointer to bit-packed secret key
+ * @param [in] rng_ctx pointer to seeded random number generator context - when
+ * pointer is non-NULL, perform a randomized signing.
+ * Otherwise use deterministic signing.
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_65_sign_ctx(struct dilithium_65_sig *sig,
+ struct dilithium_ctx *ctx,
+ const uint8_t *m, size_t mlen,
+ const struct dilithium_65_sk *sk,
+ struct crypto_rng *rng_ctx);
+
+/**
+ * @brief Initializes a signature operation
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_sign_update and dilithium_sign_final.
+ *
+ * @param [in,out] ctx pointer to an allocated Dilithium context
+ * @param [in] sk pointer to bit-packed secret key
+ *
+ * @return 0 (success) or < 0 on error; -EOPNOTSUPP is returned if a different
+ * hash than lc_shake256 is used.
+ */
+int dilithium_65_sign_init(struct dilithium_ctx *ctx,
+ const struct dilithium_65_sk *sk);
+
+/**
+ * @brief Add more data to an already initialized signature state
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_sign_init and dilithium_sign_final.
+ *
+ * @param [in] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_65_sign_update(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen);
+
+/**
+ * @brief Computes signature
+ *
+ * @param [out] sig pointer to output signature
+ * @param [in] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init and filled with
+ * dilithium_sign_update
+ * @param [in] sk pointer to bit-packed secret key
+ * @param [in] rng_ctx pointer to seeded random number generator context - when
+ * pointer is non-NULL, perform a randomized signing.
+ * Otherwise use deterministic signing.
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_65_sign_final(struct dilithium_65_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_65_sk *sk,
+ struct crypto_rng *rng_ctx);
+
+/**
+ * @brief Verifies ML-DSA signature in one shot
+ *
+ * @param [in] sig pointer to input signature
+ * @param [in] m pointer to message
+ * @param [in] mlen length of message
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 if signature could be verified correctly and -EBADMSG when
+ * signature cannot be verified, < 0 on other errors
+ */
+int dilithium_65_verify(const struct dilithium_65_sig *sig, const uint8_t *m,
+ size_t mlen, const struct dilithium_65_pk *pk);
+
+/**
+ * @brief Verifies signature with Dilithium context in one shot
+ *
+ * This API allows the caller to provide an arbitrary context buffer which
+ * is hashed together with the message to form the message digest to be signed.
+ *
+ * @param [in] sig pointer to input signature
+ * @param [in] ctx reference to the allocated Dilithium context handle
+ * @param [in] m pointer to message
+ * @param [in] mlen length of message
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 if signature could be verified correctly and -EBADMSG when
+ * signature cannot be verified, < 0 on other errors
+ */
+int dilithium_65_verify_ctx(const struct dilithium_65_sig *sig,
+ struct dilithium_ctx *ctx,
+ const uint8_t *m, size_t mlen,
+ const struct dilithium_65_pk *pk);
+
+/**
+ * @brief Initializes a signature verification operation
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_verify_update and
+ * dilithium_verify_final.
+ *
+ * @param [in,out] ctx pointer to an allocated Dilithium context
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 (success) or < 0 on error; -EOPNOTSUPP is returned if a different
+ * hash than lc_shake256 is used.
+ */
+int dilithium_65_verify_init(struct dilithium_ctx *ctx,
+ const struct dilithium_65_pk *pk);
+
+/**
+ * @brief Add more data to an already initialized signature state
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_verify_init and
+ * dilithium_verify_final.
+ *
+ * @param [in,out] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_65_verify_update(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen);
+
+/**
+ * @brief Verifies signature
+ *
+ * @param [in] sig pointer to output signature
+ * @param [in] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init and filled with
+ * dilithium_sign_update
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 if signature could be verified correctly and -EBADMSG when
+ * signature cannot be verified, < 0 on other errors
+ */
+int dilithium_65_verify_final(const struct dilithium_65_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_65_pk *pk);
+
+#endif /* __ASSEMBLER__ */
+
+#endif /* DILITHIUM_65_H */
diff --git a/crypto/ml_dsa/dilithium_87.c b/crypto/ml_dsa/dilithium_87.c
new file mode 100644
index 000000000000..fcc3e0229ed9
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_87.c
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+#define DILITHIUM_TYPE_87
+#include "dilithium.h"
+
+#define DILITHIUM_TYPE "87"
+#define DILITHIUM_MODE 5
+#define DILITHIUM_NIST_CATEGORY DILITHIUM87_NIST_CATEGORY
+#define DILITHIUM_LAMBDA DILITHIUM87_LAMBDA
+#define DILITHIUM_K DILITHIUM87_K
+#define DILITHIUM_L DILITHIUM87_L
+#define DILITHIUM_ETA DILITHIUM87_ETA
+#define DILITHIUM_TAU DILITHIUM87_TAU
+#define DILITHIUM_BETA DILITHIUM87_BETA
+#define DILITHIUM_GAMMA1 DILITHIUM87_GAMMA1
+#define DILITHIUM_GAMMA2 DILITHIUM87_GAMMA2
+#define DILITHIUM_OMEGA DILITHIUM87_OMEGA
+
+#define DILITHIUM_CTILDE_BYTES DILITHIUM87_CTILDE_BYTES
+#define DILITHIUM_POLYT1_PACKEDBYTES DILITHIUM87_POLYT1_PACKEDBYTES
+#define DILITHIUM_POLYT0_PACKEDBYTES DILITHIUM87_POLYT0_PACKEDBYTES
+#define DILITHIUM_POLYVECH_PACKEDBYTES DILITHIUM87_POLYVECH_PACKEDBYTES
+#define DILITHIUM_POLYZ_PACKEDBYTES DILITHIUM87_POLYZ_PACKEDBYTES
+#define DILITHIUM_POLYW1_PACKEDBYTES DILITHIUM87_POLYW1_PACKEDBYTES
+#define DILITHIUM_POLYETA_PACKEDBYTES DILITHIUM87_POLYETA_PACKEDBYTES
+
+#define DILITHIUM_PUBLICKEYBYTES DILITHIUM87_PUBLICKEYBYTES
+#define DILITHIUM_SECRETKEYBYTES DILITHIUM87_SECRETKEYBYTES
+#define DILITHIUM_CRYPTO_BYTES DILITHIUM87_CRYPTO_BYTES
+
+#include "dilithium_ntt.c"
+#include "dilithium_poly.c"
+#include "dilithium_rounding.c"
+#include "dilithium_signature_c.c"
diff --git a/crypto/ml_dsa/dilithium_87.h b/crypto/ml_dsa/dilithium_87.h
new file mode 100644
index 000000000000..eaed8e675383
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_87.h
@@ -0,0 +1,377 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#ifndef DILITHIUM_87_H
+#define DILITHIUM_87_H
+
+/*
+ * Dilithium Security Levels
+ * 2 -> 192 bits of security strength
+ * 3 -> 225 bits of security strength
+ * 5 -> 257 bits of security strength
+ */
+
+/* DILITHIUM_MODE 5 */
+#define DILITHIUM87_NIST_CATEGORY 5
+#define DILITHIUM87_LAMBDA 256
+#define DILITHIUM87_K 8
+#define DILITHIUM87_L 7
+#define DILITHIUM87_ETA 2
+#define DILITHIUM87_TAU 60
+#define DILITHIUM87_BETA 120
+#define DILITHIUM87_GAMMA1 (1 << 19)
+#define DILITHIUM87_GAMMA2 ((DILITHIUM_Q - 1) / 32)
+#define DILITHIUM87_OMEGA 75
+
+#define DILITHIUM87_CTILDE_BYTES (DILITHIUM87_LAMBDA * 2 / 8)
+#define DILITHIUM87_POLYT1_PACKEDBYTES 320
+#define DILITHIUM87_POLYT0_PACKEDBYTES 416
+#define DILITHIUM87_POLYVECH_PACKEDBYTES (DILITHIUM87_OMEGA + DILITHIUM87_K)
+
+#if DILITHIUM87_GAMMA1 == (1 << 17)
+#define DILITHIUM87_POLYZ_PACKEDBYTES 576
+#elif DILITHIUM87_GAMMA1 == (1 << 19)
+#define DILITHIUM87_POLYZ_PACKEDBYTES 640
+#endif
+
+#if DILITHIUM87_GAMMA2 == (DILITHIUM_Q - 1) / 88
+#define DILITHIUM87_POLYW1_PACKEDBYTES 192
+#elif DILITHIUM87_GAMMA2 == (DILITHIUM_Q - 1) / 32
+#define DILITHIUM87_POLYW1_PACKEDBYTES 128
+#endif
+
+#if DILITHIUM87_ETA == 2
+#define DILITHIUM87_POLYETA_PACKEDBYTES 96
+#elif DILITHIUM87_ETA == 4
+#define DILITHIUM87_POLYETA_PACKEDBYTES 128
+#endif
+
+/*
+ * Sizes of the different Dilithium buffer types.
+ *
+ * WARNING: Do not use these defines in your code. If you need the sizes of
+ * the different variable sizes, use sizeof of the different variable structs or
+ * use the different *_size functions documented below to retrieve the data size
+ * of a particular Dilithium component.
+ */
+#define DILITHIUM87_PUBLICKEYBYTES \
+ (DILITHIUM_SEEDBYTES + \
+ DILITHIUM87_K * DILITHIUM87_POLYT1_PACKEDBYTES)
+#define DILITHIUM87_SECRETKEYBYTES \
+ (2 * DILITHIUM_SEEDBYTES + DILITHIUM_TRBYTES + \
+ DILITHIUM87_L * DILITHIUM87_POLYETA_PACKEDBYTES + \
+ DILITHIUM87_K * DILITHIUM87_POLYETA_PACKEDBYTES + \
+ DILITHIUM87_K * DILITHIUM87_POLYT0_PACKEDBYTES)
+
+#define DILITHIUM87_CRYPTO_BYTES \
+ (DILITHIUM87_CTILDE_BYTES + \
+ DILITHIUM87_L * DILITHIUM87_POLYZ_PACKEDBYTES + \
+ DILITHIUM87_POLYVECH_PACKEDBYTES)
+
+#ifndef __ASSEMBLER__
+/**
+ * @brief Dilithium secret key
+ */
+struct dilithium_87_sk {
+ uint8_t sk[DILITHIUM87_SECRETKEYBYTES];
+};
+
+/**
+ * @brief Dilithium public key
+ */
+struct dilithium_87_pk {
+ uint8_t pk[DILITHIUM87_PUBLICKEYBYTES];
+};
+
+/**
+ * @brief Dilithium signature
+ */
+struct dilithium_87_sig {
+ uint8_t sig[DILITHIUM87_CRYPTO_BYTES];
+};
+
+/*
+ * The alignment is based on largest alignment of a polyvecl typedef - this is
+ * the AVX2 definition.
+ */
+#define DILITHIUM_AHAT_ALIGNMENT (32)
+
+/* Size of the AHat matrix for ML-DSA 87 */
+#define DILITHIUM_87_AHAT_SIZE \
+ (256 * sizeof(int32_t) * DILITHIUM87_K * DILITHIUM87_L)
+
+/**
+ * @brief Zeroize Dilithium context allocated with
+ * DILITHIUM87_CTX_ON_STACK dilithium_ed25519_alloc
+ *
+ * @param [in] ctx Dilithium context to be zeroized
+ */
+static inline void dilithium_87_ctx_zero(struct dilithium_ctx *ctx)
+{
+ if (!ctx)
+ return;
+ dilithium_hash_clear(ctx);
+ if (ctx->ahat) {
+ memzero_explicit(ctx->ahat, ctx->ahat_size);
+ ctx->ahat_expanded = 0;
+ }
+}
+
+/**
+ * @brief Allocate Dilithium stream context on heap
+ *
+ * @param [out] ctx Allocated Dilithium stream context
+ *
+ * @return: 0 on success, < 0 on error
+ */
+struct dilithium_ctx *dilithium_87_ctx_alloc(void);
+
+/**
+ * @brief Allocate Dilithium stream context on heap including additional
+ * parameter relevant for the signature operation.
+ *
+ * \note See \p DILITHIUM_87_CTX_ON_STACK_AHAT for details.
+ *
+ * @param [out] ctx Allocated Dilithium stream context
+ *
+ * @return: 0 on success, < 0 on error
+ */
+struct dilithium_ctx *dilithium_87_ctx_alloc_ahat(void);
+
+/**
+ * @brief Zeroize and free Dilithium stream context
+ *
+ * @param [in] ctx Dilithium stream context to be zeroized and freed
+ */
+void dilithium_87_ctx_zero_free(struct dilithium_ctx *ctx);
+
+/**
+ * @brief Return the size of the Dilithium secret key.
+ */
+__pure
+static inline unsigned int dilithium_87_sk_size(void)
+{
+ return sizeof_field(struct dilithium_87_sk, sk);
+}
+
+/**
+ * @brief Return the size of the Dilithium public key.
+ */
+__pure
+static inline unsigned int dilithium_87_pk_size(void)
+{
+ return sizeof_field(struct dilithium_87_pk, pk);
+}
+
+/**
+ * @brief Return the size of the Dilithium signature.
+ */
+__pure
+static inline unsigned int dilithium_87_sig_size(void)
+{
+ return sizeof_field(struct dilithium_87_sig, sig);
+}
+
+/**
+ * @brief Computes ML-DSA signature in one shot
+ *
+ * @param [out] sig pointer to output signature
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ * @param [in] sk pointer to bit-packed secret key
+ * @param [in] rng_ctx pointer to seeded random number generator context - when
+ * pointer is non-NULL, perform a randomized signing.
+ * Otherwise use deterministic signing.
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_87_sign(struct dilithium_87_sig *sig, const uint8_t *m,
+ size_t mlen, const struct dilithium_87_sk *sk,
+ struct crypto_rng *rng_ctx);
+
+/**
+ * @brief Computes signature with Dilithium context in one shot
+ *
+ * This API allows the caller to provide an arbitrary context buffer which
+ * is hashed together with the message to form the message digest to be signed.
+ *
+ * @param [out] sig pointer to output signature
+ * @param [in] ctx reference to the allocated Dilithium context handle
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ * @param [in] sk pointer to bit-packed secret key
+ * @param [in] rng_ctx pointer to seeded random number generator context - when
+ * pointer is non-NULL, perform a randomized signing.
+ * Otherwise use deterministic signing.
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_87_sign_ctx(struct dilithium_87_sig *sig,
+ struct dilithium_ctx *ctx,
+ const uint8_t *m, size_t mlen,
+ const struct dilithium_87_sk *sk,
+ struct crypto_rng *rng_ctx);
+
+/**
+ * @brief Initializes a signature operation
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_sign_update and dilithium_sign_final.
+ *
+ * @param [in,out] ctx pointer to an allocated Dilithium context
+ * @param [in] sk pointer to bit-packed secret key
+ *
+ * @return 0 (success) or < 0 on error; -EOPNOTSUPP is returned if a different
+ * hash than lc_shake256 is used.
+ */
+int dilithium_87_sign_init(struct dilithium_ctx *ctx,
+ const struct dilithium_87_sk *sk);
+
+/**
+ * @brief Add more data to an already initialized signature state
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_sign_init and dilithium_sign_final.
+ *
+ * @param [in] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_87_sign_update(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen);
+
+/**
+ * @brief Computes signature
+ *
+ * @param [out] sig pointer to output signature
+ * @param [in] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init and filled with
+ * dilithium_sign_update
+ * @param [in] sk pointer to bit-packed secret key
+ * @param [in] rng_ctx pointer to seeded random number generator context - when
+ * pointer is non-NULL, perform a randomized signing.
+ * Otherwise use deterministic signing.
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_87_sign_final(struct dilithium_87_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_87_sk *sk,
+ struct crypto_rng *rng_ctx);
+
+/**
+ * @brief Verifies ML-DSA signature in one shot
+ *
+ * @param [in] sig pointer to input signature
+ * @param [in] m pointer to message
+ * @param [in] mlen length of message
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 if signature could be verified correctly and -EBADMSG when
+ * signature cannot be verified, < 0 on other errors
+ */
+int dilithium_87_verify(const struct dilithium_87_sig *sig, const uint8_t *m,
+ size_t mlen, const struct dilithium_87_pk *pk);
+
+/**
+ * @brief Verifies signature with Dilithium context in one shot
+ *
+ * This API allows the caller to provide an arbitrary context buffer which
+ * is hashed together with the message to form the message digest to be signed.
+ *
+ * @param [in] sig pointer to input signature
+ * @param [in] ctx reference to the allocated Dilithium context handle
+ * @param [in] m pointer to message
+ * @param [in] mlen length of message
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 if signature could be verified correctly and -EBADMSG when
+ * signature cannot be verified, < 0 on other errors
+ */
+int dilithium_87_verify_ctx(const struct dilithium_87_sig *sig,
+ struct dilithium_ctx *ctx,
+ const uint8_t *m, size_t mlen,
+ const struct dilithium_87_pk *pk);
+
+/**
+ * @brief Initializes a signature verification operation
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_verify_update and
+ * dilithium_verify_final.
+ *
+ * @param [in,out] ctx pointer to an allocated Dilithium context
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 (success) or < 0 on error; -EOPNOTSUPP is returned if a different
+ * hash than lc_shake256 is used.
+ */
+int dilithium_87_verify_init(struct dilithium_ctx *ctx,
+ const struct dilithium_87_pk *pk);
+
+/**
+ * @brief Add more data to an already initialized signature state
+ *
+ * This call is intended to support messages that are located in non-contiguous
+ * places and even becomes available at different times. This call is to be
+ * used together with the dilithium_verify_init and
+ * dilithium_verify_final.
+ *
+ * @param [in,out] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init
+ * @param [in] m pointer to message to be signed
+ * @param [in] mlen length of message
+ *
+ * @return 0 (success) or < 0 on error
+ */
+int dilithium_87_verify_update(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen);
+
+/**
+ * @brief Verifies signature
+ *
+ * @param [in] sig pointer to output signature
+ * @param [in] ctx pointer to Dilithium context that was initialized with
+ * dilithium_sign_init and filled with
+ * dilithium_sign_update
+ * @param [in] pk pointer to bit-packed public key
+ *
+ * @return 0 if signature could be verified correctly and -EBADMSG when
+ * signature cannot be verified, < 0 on other errors
+ */
+int dilithium_87_verify_final(const struct dilithium_87_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_87_pk *pk);
+
+#endif /* __ASSEMBLER__ */
+
+#endif /* DILITHIUM_87_H */
diff --git a/crypto/ml_dsa/dilithium_api.c b/crypto/ml_dsa/dilithium_api.c
new file mode 100644
index 000000000000..363984f01169
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_api.c
@@ -0,0 +1,729 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#ifndef DILITHIUM_API_H
+#define DILITHIUM_API_H
+
+#include <linux/export.h>
+#include "dilithium.h"
+
+void dilithium_ctx_zero(struct dilithium_ctx *ctx)
+{
+ if (!ctx)
+ return;
+
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ dilithium_87_ctx_zero(ctx);
+#elif defined(CONFIG_CRYPTO_DILITHIUM_65)
+ dilithium_65_ctx_zero(ctx);
+#elif defined(CONFIG_CRYPTO_DILITHIUM_44)
+ dilithium_44_ctx_zero(ctx);
+#endif
+}
+EXPORT_SYMBOL(dilithium_ctx_zero);
+
+void dilithium_ctx_internal(struct dilithium_ctx *ctx)
+{
+ if (ctx)
+ ctx->ml_dsa_internal = 1;
+}
+EXPORT_SYMBOL(dilithium_ctx_internal);
+
+void dilithium_ctx_hash(struct dilithium_ctx *ctx,
+ struct crypto_shash *hash)
+{
+ if (ctx)
+ ctx->dilithium_prehash_type = hash;
+}
+EXPORT_SYMBOL(dilithium_ctx_hash);
+
+void dilithium_ctx_userctx(struct dilithium_ctx *ctx, const uint8_t *userctx,
+ size_t userctxlen)
+{
+ if (ctx) {
+ ctx->userctx = userctx;
+ ctx->userctxlen = userctxlen;
+ }
+}
+EXPORT_SYMBOL(dilithium_ctx_userctx);
+
+void dilithium_ctx_external_mu(struct dilithium_ctx *ctx, const uint8_t *external_mu,
+ size_t external_mu_len)
+{
+ if (ctx) {
+ ctx->external_mu = external_mu;
+ ctx->external_mu_len = external_mu_len;
+ }
+}
+EXPORT_SYMBOL(dilithium_ctx_external_mu);
+
+void dilithium_ctx_drop_ahat(struct dilithium_ctx *ctx)
+{
+ if (ctx)
+ ctx->ahat_expanded = 0;
+}
+EXPORT_SYMBOL(dilithium_ctx_drop_ahat);
+
+enum dilithium_type dilithium_sk_type(const struct dilithium_sk *sk)
+{
+ if (!sk)
+ return DILITHIUM_UNKNOWN;
+ return sk->dilithium_type;
+}
+EXPORT_SYMBOL(dilithium_sk_type);
+
+enum dilithium_type dilithium_pk_type(const struct dilithium_pk *pk)
+{
+ if (!pk)
+ return DILITHIUM_UNKNOWN;
+ return pk->dilithium_type;
+}
+EXPORT_SYMBOL(dilithium_pk_type);
+
+enum dilithium_type dilithium_sig_type(const struct dilithium_sig *sig)
+{
+ if (!sig)
+ return DILITHIUM_UNKNOWN;
+ return sig->dilithium_type;
+}
+EXPORT_SYMBOL(dilithium_sig_type);
+
+__pure unsigned int dilithium_sk_size(enum dilithium_type dilithium_type)
+{
+ switch (dilithium_type) {
+ case DILITHIUM_87:
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ return sizeof_field(struct dilithium_sk, key.sk_87);
+#else
+ return 0;
+#endif
+ case DILITHIUM_65:
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ return sizeof_field(struct dilithium_sk, key.sk_65);
+#else
+ return 0;
+#endif
+ case DILITHIUM_44:
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ return sizeof_field(struct dilithium_sk, key.sk_44);
+#else
+ return 0;
+#endif
+ case DILITHIUM_UNKNOWN:
+ default:
+ return 0;
+ }
+}
+EXPORT_SYMBOL(dilithium_sk_size);
+
+__pure unsigned int dilithium_pk_size(enum dilithium_type dilithium_type)
+{
+ switch (dilithium_type) {
+ case DILITHIUM_87:
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ return sizeof_field(struct dilithium_pk, key.pk_87);
+#else
+ return 0;
+#endif
+ case DILITHIUM_65:
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ return sizeof_field(struct dilithium_pk, key.pk_65);
+#else
+ return 0;
+#endif
+ case DILITHIUM_44:
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ return sizeof_field(struct dilithium_pk, key.pk_44);
+#else
+ return 0;
+#endif
+ case DILITHIUM_UNKNOWN:
+ default:
+ return 0;
+ }
+}
+EXPORT_SYMBOL(dilithium_pk_size);
+
+__pure unsigned int dilithium_sig_size(enum dilithium_type dilithium_type)
+{
+ switch (dilithium_type) {
+ case DILITHIUM_87:
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ return sizeof_field(struct dilithium_sig, sig.sig_87);
+#else
+ return 0;
+#endif
+ case DILITHIUM_65:
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ return sizeof_field(struct dilithium_sig, sig.sig_65);
+#else
+ return 0;
+#endif
+ case DILITHIUM_44:
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ return sizeof_field(struct dilithium_sig, sig.sig_44);
+#else
+ return 0;
+#endif
+ case DILITHIUM_UNKNOWN:
+ default:
+ return 0;
+ }
+}
+EXPORT_SYMBOL(dilithium_sig_size);
+
+int dilithium_sk_load(struct dilithium_sk *sk,
+ const uint8_t *src_key, size_t src_key_len)
+{
+ if (!sk || !src_key || src_key_len == 0) {
+ return -EINVAL;
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ } else if (src_key_len == dilithium_sk_size(DILITHIUM_87)) {
+ struct dilithium_87_sk *_sk = &sk->key.sk_87;
+
+ memcpy(_sk->sk, src_key, src_key_len);
+ sk->dilithium_type = DILITHIUM_87;
+ return 0;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ } else if (src_key_len == dilithium_sk_size(DILITHIUM_65)) {
+ struct dilithium_65_sk *_sk = &sk->key.sk_65;
+
+ memcpy(_sk->sk, src_key, src_key_len);
+ sk->dilithium_type = DILITHIUM_65;
+ return 0;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ } else if (src_key_len == dilithium_sk_size(DILITHIUM_44)) {
+ struct dilithium_44_sk *_sk = &sk->key.sk_44;
+
+ memcpy(_sk->sk, src_key, src_key_len);
+ sk->dilithium_type = DILITHIUM_44;
+ return 0;
+#endif
+ } else {
+ return -EINVAL;
+ }
+}
+EXPORT_SYMBOL(dilithium_sk_load);
+
+int dilithium_pk_load(struct dilithium_pk *pk,
+ const uint8_t *src_key, size_t src_key_len)
+{
+ if (!pk || !src_key || src_key_len == 0) {
+ return -EINVAL;
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ } else if (src_key_len == dilithium_pk_size(DILITHIUM_87)) {
+ struct dilithium_87_pk *_pk = &pk->key.pk_87;
+
+ memcpy(_pk->pk, src_key, src_key_len);
+ pk->dilithium_type = DILITHIUM_87;
+ return 0;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ } else if (src_key_len == dilithium_pk_size(DILITHIUM_65)) {
+ struct dilithium_65_pk *_pk = &pk->key.pk_65;
+
+ memcpy(_pk->pk, src_key, src_key_len);
+ pk->dilithium_type = DILITHIUM_65;
+ return 0;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ } else if (src_key_len == dilithium_pk_size(DILITHIUM_44)) {
+ struct dilithium_44_pk *_pk = &pk->key.pk_44;
+
+ memcpy(_pk->pk, src_key, src_key_len);
+ pk->dilithium_type = DILITHIUM_44;
+ return 0;
+#endif
+ } else {
+ return -EINVAL;
+ }
+}
+EXPORT_SYMBOL(dilithium_pk_load);
+
+int dilithium_sig_load(struct dilithium_sig *sig,
+ const uint8_t *src_sig, size_t src_sig_len)
+{
+ if (!sig || !src_sig || src_sig_len == 0) {
+ return -EINVAL;
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ } else if (src_sig_len == dilithium_sig_size(DILITHIUM_87)) {
+ struct dilithium_87_sig *_sig = &sig->sig.sig_87;
+
+ memcpy(_sig->sig, src_sig, src_sig_len);
+ sig->dilithium_type = DILITHIUM_87;
+ return 0;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ } else if (src_sig_len == dilithium_sig_size(DILITHIUM_65)) {
+ struct dilithium_65_sig *_sig = &sig->sig.sig_65;
+
+ memcpy(_sig->sig, src_sig, src_sig_len);
+ sig->dilithium_type = DILITHIUM_65;
+ return 0;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ } else if (src_sig_len == dilithium_sig_size(DILITHIUM_44)) {
+ struct dilithium_44_sig *_sig = &sig->sig.sig_44;
+
+ memcpy(_sig->sig, src_sig, src_sig_len);
+ sig->dilithium_type = DILITHIUM_44;
+ return 0;
+#endif
+ } else {
+ return -EINVAL;
+ }
+}
+EXPORT_SYMBOL(dilithium_sig_load);
+
+int dilithium_sk_ptr(uint8_t **dilithium_key,
+ size_t *dilithium_key_len, struct dilithium_sk *sk)
+{
+ if (!sk || !dilithium_key || !dilithium_key_len) {
+ return -EINVAL;
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ } else if (sk->dilithium_type == DILITHIUM_87) {
+ struct dilithium_87_sk *_sk = &sk->key.sk_87;
+
+ *dilithium_key = _sk->sk;
+ *dilithium_key_len = dilithium_sk_size(sk->dilithium_type);
+ return 0;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ } else if (sk->dilithium_type == DILITHIUM_65) {
+ struct dilithium_65_sk *_sk = &sk->key.sk_65;
+
+ *dilithium_key = _sk->sk;
+ *dilithium_key_len = dilithium_sk_size(sk->dilithium_type);
+ return 0;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ } else if (sk->dilithium_type == DILITHIUM_44) {
+ struct dilithium_44_sk *_sk = &sk->key.sk_44;
+
+ *dilithium_key = _sk->sk;
+ *dilithium_key_len = dilithium_sk_size(sk->dilithium_type);
+ return 0;
+#endif
+ } else {
+ return -EINVAL;
+ }
+}
+EXPORT_SYMBOL(dilithium_sk_ptr);
+
+int dilithium_pk_ptr(uint8_t **dilithium_key,
+ size_t *dilithium_key_len, struct dilithium_pk *pk)
+{
+ if (!pk || !dilithium_key || !dilithium_key_len) {
+ return -EINVAL;
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ } else if (pk->dilithium_type == DILITHIUM_87) {
+ struct dilithium_87_pk *_pk = &pk->key.pk_87;
+
+ *dilithium_key = _pk->pk;
+ *dilithium_key_len = dilithium_pk_size(pk->dilithium_type);
+ return 0;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ } else if (pk->dilithium_type == DILITHIUM_65) {
+ struct dilithium_65_pk *_pk = &pk->key.pk_65;
+
+ *dilithium_key = _pk->pk;
+ *dilithium_key_len = dilithium_pk_size(pk->dilithium_type);
+ return 0;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ } else if (pk->dilithium_type == DILITHIUM_44) {
+ struct dilithium_44_pk *_pk = &pk->key.pk_44;
+
+ *dilithium_key = _pk->pk;
+ *dilithium_key_len = dilithium_pk_size(pk->dilithium_type);
+ return 0;
+#endif
+ } else {
+ return -EINVAL;
+ }
+}
+EXPORT_SYMBOL(dilithium_pk_ptr);
+
+int dilithium_sig_ptr(uint8_t **dilithium_sig,
+ size_t *dilithium_sig_len, struct dilithium_sig *sig)
+{
+ if (!sig || !dilithium_sig || !dilithium_sig_len) {
+ return -EINVAL;
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ } else if (sig->dilithium_type == DILITHIUM_87) {
+ struct dilithium_87_sig *_sig = &sig->sig.sig_87;
+
+ *dilithium_sig = _sig->sig;
+ *dilithium_sig_len = dilithium_sig_size(sig->dilithium_type);
+ return 0;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ } else if (sig->dilithium_type == DILITHIUM_65) {
+ struct dilithium_65_sig *_sig = &sig->sig.sig_65;
+
+ *dilithium_sig = _sig->sig;
+ *dilithium_sig_len = dilithium_sig_size(sig->dilithium_type);
+ return 0;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ } else if (sig->dilithium_type == DILITHIUM_44) {
+ struct dilithium_44_sig *_sig = &sig->sig.sig_44;
+
+ *dilithium_sig = _sig->sig;
+ *dilithium_sig_len = dilithium_sig_size(sig->dilithium_type);
+ return 0;
+#endif
+ } else {
+ return -EINVAL;
+ }
+}
+EXPORT_SYMBOL(dilithium_sig_ptr);
+
+int dilithium_sign(struct dilithium_sig *sig,
+ const uint8_t *m, size_t mlen,
+ const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx)
+{
+ if (!sk || !sig)
+ return -EINVAL;
+
+ switch (sk->dilithium_type) {
+ case DILITHIUM_87:
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ sig->dilithium_type = DILITHIUM_87;
+ return dilithium_87_sign(&sig->sig.sig_87, m, mlen,
+ &sk->key.sk_87, rng_ctx);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_65:
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ sig->dilithium_type = DILITHIUM_65;
+ return dilithium_65_sign(&sig->sig.sig_65, m, mlen,
+ &sk->key.sk_65, rng_ctx);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_44:
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ sig->dilithium_type = DILITHIUM_44;
+ return dilithium_44_sign(&sig->sig.sig_44, m, mlen,
+ &sk->key.sk_44, rng_ctx);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_UNKNOWN:
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+EXPORT_SYMBOL(dilithium_sign);
+
+int dilithium_sign_ctx(struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen, const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx)
+{
+ if (!sk || !sig)
+ return -EINVAL;
+
+ switch (sk->dilithium_type) {
+ case DILITHIUM_87:
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ sig->dilithium_type = DILITHIUM_87;
+ return dilithium_87_sign_ctx(&sig->sig.sig_87, ctx, m, mlen,
+ &sk->key.sk_87, rng_ctx);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_65:
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ sig->dilithium_type = DILITHIUM_65;
+ return dilithium_65_sign_ctx(&sig->sig.sig_65, ctx, m, mlen,
+ &sk->key.sk_65, rng_ctx);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_44:
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ sig->dilithium_type = DILITHIUM_44;
+ return dilithium_44_sign_ctx(&sig->sig.sig_44, ctx, m, mlen,
+ &sk->key.sk_44, rng_ctx);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_UNKNOWN:
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+EXPORT_SYMBOL(dilithium_sign_ctx);
+
+int dilithium_sign_init(struct dilithium_ctx *ctx,
+ const struct dilithium_sk *sk)
+{
+ if (!sk)
+ return -EINVAL;
+
+ switch (sk->dilithium_type) {
+ case DILITHIUM_87:
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ return dilithium_87_sign_init(ctx, &sk->key.sk_87);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_65:
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ return dilithium_65_sign_init(ctx, &sk->key.sk_65);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_44:
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ return dilithium_44_sign_init(ctx, &sk->key.sk_44);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_UNKNOWN:
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+EXPORT_SYMBOL(dilithium_sign_init);
+
+int dilithium_sign_update(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen)
+{
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ return dilithium_87_sign_update(ctx, m, mlen);
+#elif defined(CONFIG_CRYPTO_DILITHIUM_65)
+ return dilithium_65_sign_update(ctx, m, mlen);
+#elif defined(CONFIG_CRYPTO_DILITHIUM_44)
+ return dilithium_44_sign_update(ctx, m, mlen);
+#else
+ return -EOPNOTSUPP;
+#endif
+}
+EXPORT_SYMBOL(dilithium_sign_update);
+
+int dilithium_sign_final(struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx)
+{
+ if (!sk || !sig)
+ return -EINVAL;
+
+ switch (sk->dilithium_type) {
+ case DILITHIUM_87:
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ sig->dilithium_type = DILITHIUM_87;
+ return dilithium_87_sign_final(&sig->sig.sig_87, ctx,
+ &sk->key.sk_87, rng_ctx);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_65:
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ sig->dilithium_type = DILITHIUM_65;
+ return dilithium_65_sign_final(&sig->sig.sig_65, ctx,
+ &sk->key.sk_65, rng_ctx);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_44:
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ sig->dilithium_type = DILITHIUM_44;
+ return dilithium_44_sign_final(&sig->sig.sig_44, ctx,
+ &sk->key.sk_44, rng_ctx);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_UNKNOWN:
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+EXPORT_SYMBOL(dilithium_sign_final);
+
+int dilithium_verify(const struct dilithium_sig *sig, const uint8_t *m,
+ size_t mlen, const struct dilithium_pk *pk)
+{
+ if (!pk || !sig || sig->dilithium_type != pk->dilithium_type)
+ return -EINVAL;
+
+ switch (pk->dilithium_type) {
+ case DILITHIUM_87:
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ return dilithium_87_verify(&sig->sig.sig_87, m, mlen,
+ &pk->key.pk_87);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_65:
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ return dilithium_65_verify(&sig->sig.sig_65, m, mlen,
+ &pk->key.pk_65);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_44:
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ return dilithium_44_verify(&sig->sig.sig_44, m, mlen,
+ &pk->key.pk_44);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_UNKNOWN:
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+EXPORT_SYMBOL(dilithium_verify);
+
+int dilithium_verify_ctx(const struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen, const struct dilithium_pk *pk)
+{
+ if (!pk || !sig || sig->dilithium_type != pk->dilithium_type)
+ return -EINVAL;
+
+ switch (pk->dilithium_type) {
+ case DILITHIUM_87:
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ return dilithium_87_verify_ctx(&sig->sig.sig_87, ctx, m,
+ mlen, &pk->key.pk_87);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_65:
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ return dilithium_65_verify_ctx(&sig->sig.sig_65, ctx, m,
+ mlen, &pk->key.pk_65);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_44:
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ return dilithium_44_verify_ctx(&sig->sig.sig_44, ctx, m,
+ mlen, &pk->key.pk_44);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_UNKNOWN:
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+EXPORT_SYMBOL(dilithium_verify_ctx);
+
+int dilithium_verify_init(struct dilithium_ctx *ctx,
+ const struct dilithium_pk *pk)
+{
+ if (!pk)
+ return -EINVAL;
+
+ switch (pk->dilithium_type) {
+ case DILITHIUM_87:
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ return dilithium_87_verify_init(ctx, &pk->key.pk_87);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_65:
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ return dilithium_65_verify_init(ctx, &pk->key.pk_65);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_44:
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ return dilithium_44_verify_init(ctx, &pk->key.pk_44);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_UNKNOWN:
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+EXPORT_SYMBOL(dilithium_verify_init);
+
+int dilithium_verify_update(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen)
+{
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ return dilithium_87_verify_update(ctx, m, mlen);
+#elif defined(CONFIG_CRYPTO_DILITHIUM_65)
+ return dilithium_65_verify_update(ctx, m, mlen);
+#elif defined(CONFIG_CRYPTO_DILITHIUM_44)
+ return dilithium_44_verify_update(ctx, m, mlen);
+#else
+ return -EOPNOTSUPP;
+#endif
+}
+EXPORT_SYMBOL(dilithium_verify_update);
+
+int dilithium_verify_final(const struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_pk *pk)
+{
+ if (!pk || !sig || sig->dilithium_type != pk->dilithium_type)
+ return -EINVAL;
+
+ switch (pk->dilithium_type) {
+ case DILITHIUM_87:
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ return dilithium_87_verify_final(&sig->sig.sig_87, ctx,
+ &pk->key.pk_87);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_65:
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ return dilithium_65_verify_final(&sig->sig.sig_65, ctx,
+ &pk->key.pk_65);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_44:
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ return dilithium_44_verify_final(&sig->sig.sig_44, ctx,
+ &pk->key.pk_44);
+#else
+ return -EOPNOTSUPP;
+#endif
+ case DILITHIUM_UNKNOWN:
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+EXPORT_SYMBOL(dilithium_verify_final);
+
+#endif /* DILITHIUM_API_H */
diff --git a/crypto/ml_dsa/dilithium_debug.h b/crypto/ml_dsa/dilithium_debug.h
new file mode 100644
index 000000000000..9448c252175f
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_debug.h
@@ -0,0 +1,80 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2023 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+
+#ifndef DILITHIUM_DEBUG_H
+#define DILITHIUM_DEBUG_H
+
+#include "dilithium_type.h"
+
+#ifdef DILITHIUM_DEBUG
+
+/* Disable selftests */
+#define DILITHIUM_TEST_INIT 1
+
+void dilithium_print_buffer(const uint8_t *buffer, const size_t bufferlen,
+ const char *explanation);
+void dilithium_print_polyvecl_k(polyvecl mat[DILITHIUM_K],
+ const char *explanation);
+void dilithium_print_polyvecl(polyvecl *polyvec, const char *explanation);
+void dilithium_print_polyveck(polyveck *polyvec, const char *explanation);
+void dilithium_print_poly(poly *vec, const char *explanation);
+
+#else /* DILITHIUM_DEBUG */
+
+/* Enable selftests */
+#define DILITHIUM_TEST_INIT 0
+
+static inline void dilithium_print_buffer(const uint8_t *buffer,
+ const size_t bufferlen,
+ const char *explanation)
+{
+ (void)buffer;
+ (void)bufferlen;
+ (void)explanation;
+}
+
+static inline void dilithium_print_polyvecl_k(polyvecl mat[DILITHIUM_K],
+ const char *explanation)
+{
+ (void)mat;
+ (void)explanation;
+}
+
+static inline void dilithium_print_polyvecl(polyvecl *polyvec,
+ const char *explanation)
+{
+ (void)polyvec;
+ (void)explanation;
+}
+
+static inline void dilithium_print_polyveck(polyveck *polyvec,
+ const char *explanation)
+{
+ (void)polyvec;
+ (void)explanation;
+}
+
+static inline void dilithium_print_poly(poly *vec, const char *explanation)
+{
+ (void)vec;
+ (void)explanation;
+}
+
+#endif /* DILITHIUM_DEBUG */
+
+#endif /* DILITHIUM_DEBUG_H */
diff --git a/crypto/ml_dsa/dilithium_ntt.c b/crypto/ml_dsa/dilithium_ntt.c
new file mode 100644
index 000000000000..839f4cb8a0aa
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_ntt.c
@@ -0,0 +1,89 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#include "dilithium_ntt.h"
+#include "dilithium_reduce.h"
+
+/**
+ * @brief ntt - Forward NTT, in-place. No modular reduction is performed after
+ * additions or subtractions. Output vector is in bitreversed
+ * order.
+ *
+ * @param [in,out] p input/output coefficient array
+ */
+void ntt(int32_t a[DILITHIUM_N])
+{
+ unsigned int len, start, j, k;
+ int32_t zeta, t;
+
+ k = 0;
+
+ for (len = 128; len > 0; len >>= 1) {
+ for (start = 0; start < DILITHIUM_N; start = j + len) {
+ zeta = dilithium_zetas[++k];
+ for (j = start; j < start + len; ++j) {
+ t = montgomery_reduce((int64_t)zeta *
+ a[j + len]);
+ a[j + len] = a[j] - t;
+ a[j] = a[j] + t;
+ }
+ }
+ }
+}
+
+/**
+ * @brief invntt_tomont - Inverse NTT and multiplication by Montgomery factor
+ * 2^32. In-place. No modular reductions after additions
+ * or subtractions; input coefficients need to be smaller
+ * than Q in absolute value. Output coefficient are
+ * smaller than Q in absolute value.
+ *
+ * @param [in,out] p input/output coefficient array
+ */
+void invntt_tomont(int32_t a[DILITHIUM_N])
+{
+ unsigned int start, len, j, k;
+ int32_t t, zeta;
+ const int32_t f = 41978; // mont^2/256
+
+ k = 256;
+
+ for (len = 1; len < DILITHIUM_N; len <<= 1) {
+ for (start = 0; start < DILITHIUM_N; start = j + len) {
+ zeta = -dilithium_zetas[--k];
+ for (j = start; j < start + len; ++j) {
+ t = a[j];
+ a[j] = t + a[j + len];
+ a[j + len] = t - a[j + len];
+ a[j + len] = montgomery_reduce((int64_t)zeta *
+ a[j + len]);
+ }
+ }
+ }
+
+ for (j = 0; j < DILITHIUM_N; ++j)
+ a[j] = montgomery_reduce((int64_t)f * a[j]);
+}
diff --git a/crypto/ml_dsa/dilithium_ntt.h b/crypto/ml_dsa/dilithium_ntt.h
new file mode 100644
index 000000000000..5dfea3a936cd
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_ntt.h
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#ifndef DILITHIUM_NTT_H
+#define DILITHIUM_NTT_H
+
+#include "dilithium_type.h"
+
+void ntt(int32_t a[DILITHIUM_N]);
+void invntt_tomont(int32_t a[DILITHIUM_N]);
+
+#endif /* DILITHIUM_NTT_H */
diff --git a/crypto/ml_dsa/dilithium_pack.h b/crypto/ml_dsa/dilithium_pack.h
new file mode 100644
index 000000000000..e17c6256a16d
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_pack.h
@@ -0,0 +1,210 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#ifndef DILITHIUM_PACK_H
+#define DILITHIUM_PACK_H
+
+/*******************************************************************************
+ * Pack / Unpack public key
+ ******************************************************************************/
+
+static inline void unpack_pk_rho(uint8_t rho[DILITHIUM_SEEDBYTES],
+ const struct dilithium_pk *pk)
+{
+ memcpy(rho, pk->pk, DILITHIUM_SEEDBYTES);
+}
+
+static inline void unpack_pk_t1(polyveck *t1, const struct dilithium_pk *pk)
+{
+ unsigned int i;
+ const uint8_t *pubkey = pk->pk + DILITHIUM_SEEDBYTES;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ polyt1_unpack(&t1->vec[i],
+ pubkey + i * DILITHIUM_POLYT1_PACKEDBYTES);
+}
+
+/*******************************************************************************
+ * Pack / Unpack secret key
+ ******************************************************************************/
+static inline void unpack_sk_key(uint8_t key[DILITHIUM_SEEDBYTES],
+ const struct dilithium_sk *sk)
+{
+ memcpy(key, sk->sk + DILITHIUM_SEEDBYTES, DILITHIUM_SEEDBYTES);
+}
+
+static inline void unpack_sk_tr(uint8_t tr[DILITHIUM_TRBYTES],
+ const struct dilithium_sk *sk)
+{
+ memcpy(tr, sk->sk + 2 * DILITHIUM_SEEDBYTES, DILITHIUM_TRBYTES);
+}
+
+static inline void unpack_sk_s1(polyvecl *s1, const struct dilithium_sk *sk)
+{
+ unsigned int i;
+ const uint8_t *seckey =
+ sk->sk + 2 * DILITHIUM_SEEDBYTES + DILITHIUM_TRBYTES;
+
+ for (i = 0; i < DILITHIUM_L; ++i)
+ polyeta_unpack(&s1->vec[i],
+ seckey + i * DILITHIUM_POLYETA_PACKEDBYTES);
+}
+
+static inline void unpack_sk_s2(polyveck *s2, const struct dilithium_sk *sk)
+{
+ unsigned int i;
+ const uint8_t *seckey =
+ sk->sk + 2 * DILITHIUM_SEEDBYTES + DILITHIUM_TRBYTES +
+ DILITHIUM_L * DILITHIUM_POLYETA_PACKEDBYTES;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ polyeta_unpack(&s2->vec[i],
+ seckey + i * DILITHIUM_POLYETA_PACKEDBYTES);
+}
+
+static inline void unpack_sk_t0(polyveck *t0, const struct dilithium_sk *sk)
+{
+ unsigned int i;
+ const uint8_t *seckey =
+ sk->sk + 2 * DILITHIUM_SEEDBYTES + DILITHIUM_TRBYTES +
+ DILITHIUM_L * DILITHIUM_POLYETA_PACKEDBYTES +
+ DILITHIUM_K * DILITHIUM_POLYETA_PACKEDBYTES;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ polyt0_unpack(&t0->vec[i],
+ seckey + i * DILITHIUM_POLYT0_PACKEDBYTES);
+}
+
+/**
+ * @brief pack_sig - Bit-pack signature sig = (c, z, h).
+ *
+ * NOTE: A signature is the concatenation of sig = (c || packed z || packed h).
+ * As c is already present in the first bytes of sig, this function does
+ * not need to copy it yet again to the right location. This implies that
+ * this function does not process c.
+ *
+ * @param [out] sig signature
+ * @param [in] z pointer to vector z
+ * @param [in] h pointer to hint vector h
+ */
+static inline void pack_sig(struct dilithium_sig *sig, const polyvecl *z,
+ const polyveck *h)
+{
+ unsigned int i, j, k;
+ /* Skip c */
+ uint8_t *signature = sig->sig + DILITHIUM_CTILDE_BYTES;
+
+ BUILD_BUG_ON((1ULL << (sizeof(j) << 3)) < DILITHIUM_N);
+ BUILD_BUG_ON((1ULL << (sizeof(k) << 3)) < DILITHIUM_N);
+
+ for (i = 0; i < DILITHIUM_L; ++i)
+ polyz_pack(signature + i * DILITHIUM_POLYZ_PACKEDBYTES,
+ &z->vec[i]);
+ signature += DILITHIUM_L * DILITHIUM_POLYZ_PACKEDBYTES;
+
+ /* Encode h */
+ memset(signature, 0, DILITHIUM_OMEGA + DILITHIUM_K);
+
+ k = 0;
+ for (i = 0; i < DILITHIUM_K; ++i) {
+ for (j = 0; j < DILITHIUM_N; ++j)
+ if (h->vec[i].coeffs[j] != 0)
+ signature[k++] = (uint8_t)j;
+
+ signature[DILITHIUM_OMEGA + i] = (uint8_t)k;
+ }
+}
+
+/**
+ * @brief unpack_sig_z - Unpack z part of signature sig = (c, z, h).
+ *
+ * NOTE: The c value is not unpacked as it can be used right from the signature.
+ * To access it, a caller simply needs to use the first
+ * DILITHIUM_CTILDE_BYTES of the signature.
+ *
+ * @param [out] z pointer to output vector z
+ * @param [in] sig signature
+ */
+static inline void unpack_sig_z(polyvecl *z, const struct dilithium_sig *sig)
+{
+ unsigned int i;
+ /* Skip c */
+ const uint8_t *signature = sig->sig + DILITHIUM_CTILDE_BYTES;
+
+ for (i = 0; i < DILITHIUM_L; ++i)
+ polyz_unpack(&z->vec[i],
+ signature + i * DILITHIUM_POLYZ_PACKEDBYTES);
+}
+
+/**
+ * @brief unpack_sig - Unpack h value of signature sig = (c, z, h).
+ *
+ * NOTE: The c value is not unpacked as it can be used right from the signature.
+ * To access it, a caller simply needs to use the first
+ * DILITHIUM_CTILDE_BYTES of the signature.
+ *
+ * @param [out] h pointer to output hint vector h
+ * @param [in] sig signature
+ *
+ * @return 1 in case of malformed signature; otherwise 0.
+ */
+static inline int unpack_sig_h(polyveck *h, const struct dilithium_sig *sig)
+{
+ unsigned int i, j, k;
+ /* Skip c */
+ const uint8_t *signature =
+ sig->sig + DILITHIUM_CTILDE_BYTES +
+ DILITHIUM_L * DILITHIUM_POLYZ_PACKEDBYTES;
+
+ /* Decode h */
+ k = 0;
+ for (i = 0; i < DILITHIUM_K; ++i) {
+ for (j = 0; j < DILITHIUM_N; ++j)
+ h->vec[i].coeffs[j] = 0;
+
+ if (signature[DILITHIUM_OMEGA + i] < k ||
+ signature[DILITHIUM_OMEGA + i] > DILITHIUM_OMEGA)
+ return 1;
+
+ for (j = k; j < signature[DILITHIUM_OMEGA + i]; ++j) {
+ /* Coefficients are ordered for strong unforgeability */
+ if (j > k && signature[j] <= signature[j - 1])
+ return 1;
+ h->vec[i].coeffs[signature[j]] = 1;
+ }
+
+ k = signature[DILITHIUM_OMEGA + i];
+ }
+
+ /* Extra indices are zero for strong unforgeability */
+ for (j = k; j < DILITHIUM_OMEGA; ++j)
+ if (signature[j])
+ return 1;
+
+ return 0;
+}
+
+#endif /* DILITHIUM_PACK_H */
diff --git a/crypto/ml_dsa/dilithium_poly.c b/crypto/ml_dsa/dilithium_poly.c
new file mode 100644
index 000000000000..45ce07d2d4fb
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_poly.c
@@ -0,0 +1,586 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#include "dilithium_poly.h"
+#include "dilithium_poly_common.h"
+#include "dilithium_service_helpers.h"
+#include <crypto/sha3.h>
+
+/**
+ * @brief poly_chknorm - Check infinity norm of polynomial against given bound.
+ * Assumes input coefficients were reduced by reduce32().
+ *
+ * @param [in] a pointer to polynomial
+ * @param [in] B norm bound
+ *
+ * @return 0 if norm is strictly smaller than B <= (Q-1)/8 and 1 otherwise.
+ */
+int poly_chknorm(const poly *a, int32_t B)
+{
+ unsigned int i;
+ int32_t t;
+
+ if (B > (DILITHIUM_Q - 1) / 8)
+ return 1;
+
+ /*
+ * It is ok to leak which coefficient violates the bound since
+ * the probability for each coefficient *is independent of secret
+ * data but we must not leak the sign of the centralized representative.
+ */
+ for (i = 0; i < DILITHIUM_N; ++i) {
+ /* Absolute value */
+ t = a->coeffs[i] >> 31;
+ t = a->coeffs[i] - (t & 2 * a->coeffs[i]);
+
+ if (t >= B)
+ return 1;
+ }
+
+ return 0;
+}
+
+/**
+ * @brief poly_uniform - Sample polynomial with uniformly random coefficients
+ * in [0,Q-1] by performing rejection sampling on the
+ * output stream of SHAKE128(seed|nonce).
+ *
+ * @param [out] a pointer to output polynomial
+ * @param [in] seed byte array with seed of length DILITHIUM_SEEDBYTES
+ * @param [in] nonce 2-byte nonce
+ */
+void poly_uniform(poly *a, const uint8_t seed[DILITHIUM_SEEDBYTES],
+ uint16_t nonce, void *ws_buf)
+{
+ struct shake128_ctx hash_ctx;
+ unsigned int i, ctr, off;
+ unsigned int buflen = POLY_UNIFORM_NBLOCKS * SHAKE128_BLOCK_SIZE;
+ uint8_t *buf = ws_buf;
+
+ shake128_init(&hash_ctx);
+ shake128_update(&hash_ctx, seed, DILITHIUM_SEEDBYTES);
+ shake128_update(&hash_ctx, (uint8_t *)&nonce, sizeof(nonce));
+ shake128_squeeze(&hash_ctx, buf, buflen);
+
+ ctr = rej_uniform(a->coeffs, DILITHIUM_N, buf, buflen);
+
+ while (ctr < DILITHIUM_N) {
+ off = buflen % 3;
+ for (i = 0; i < off; ++i)
+ buf[i] = buf[buflen - off + i];
+
+ shake128_squeeze(&hash_ctx, buf + off, SHAKE128_BLOCK_SIZE);
+ buflen = DILITHIUM_SEEDBYTES + off;
+ ctr += rej_uniform(a->coeffs + ctr, DILITHIUM_N - ctr, buf,
+ buflen);
+ }
+
+ shake128_clear(&hash_ctx);
+}
+
+/**
+ * @brief poly_uniform_eta - Sample polynomial with uniformly random
+ * coefficients in [-ETA,ETA] by performing rejection
+ * sampling on the output stream from
+ * SHAKE256(seed|nonce).
+ *
+ * @param [out] a pointer to output polynomial
+ * @param [in] seed byte array with seed of length DILITHIUM_CRHBYTES
+ * @param [in] nonce 2-byte nonce
+ */
+void poly_uniform_eta(poly *a, const uint8_t seed[DILITHIUM_CRHBYTES],
+ uint16_t nonce, void *ws_buf)
+{
+ struct shake256_ctx hash_ctx;
+ unsigned int ctr;
+ uint8_t *buf = ws_buf;
+
+ shake256_init(&hash_ctx);
+ shake256_update(&hash_ctx, seed, DILITHIUM_CRHBYTES);
+ shake256_update(&hash_ctx, (uint8_t *)&nonce, sizeof(nonce));
+ shake256_squeeze(&hash_ctx, buf, POLY_UNIFORM_ETA_BYTES);
+
+ ctr = rej_eta(a->coeffs, DILITHIUM_N, buf, POLY_UNIFORM_ETA_BYTES);
+
+ while (ctr < DILITHIUM_N) {
+ shake256_squeeze(&hash_ctx, buf, POLY_UNIFORM_ETA_BYTES);
+
+ ctr += rej_eta(a->coeffs + ctr, DILITHIUM_N - ctr, buf,
+ SHAKE256_BLOCK_SIZE);
+ }
+
+ shake256_clear(&hash_ctx);
+}
+
+/**
+ * @brief poly_uniform_gamma1 - Sample polynomial with uniformly random
+ * coefficients in [-(GAMMA1 - 1), GAMMA1] by
+ * unpacking output stream of
+ * SHAKE256(seed|nonce).
+ *
+ * @param [out] a pointer to output polynomial
+ * @param [in] seed: byte array with seed of length DILITHIUM_CRHBYTES
+ * @param nonce 16-bit nonce
+ */
+void poly_uniform_gamma1(poly *a, const uint8_t seed[DILITHIUM_CRHBYTES],
+ uint16_t nonce, void *ws_buf)
+{
+ struct shake256_ctx hash_ctx;
+
+ shake256_init(&hash_ctx);
+ shake256_update(&hash_ctx, seed, DILITHIUM_CRHBYTES);
+ shake256_update(&hash_ctx, (uint8_t *)&nonce, sizeof(nonce));
+ shake256_squeeze(&hash_ctx, ws_buf, POLY_UNIFORM_GAMMA1_BYTES);
+ shake256_clear(&hash_ctx);
+
+ polyz_unpack(a, ws_buf);
+}
+
+/**
+ * @brief poly_challenge - Implementation of H. Samples polynomial with TAU
+ * nonzero coefficients in {-1,1} using the output
+ * stream of SHAKE256(seed).
+ *
+ * @param [out] c pointer to output polynomial
+ * @param [in] mu byte array containing seed of length DILITHIUM_CTILDE_BYTES
+ */
+void poly_challenge(poly *c, const uint8_t seed[DILITHIUM_CTILDE_BYTES],
+ void *ws_buf)
+{
+ struct shake256_ctx hash_ctx;
+ unsigned int i, b, pos;
+ uint64_t signs;
+ uint8_t *buf = ws_buf;
+
+ shake256_init(&hash_ctx);
+ shake256_update(&hash_ctx, seed, DILITHIUM_CTILDE_BYTES);
+ shake256_squeeze(&hash_ctx, buf, POLY_CHALLENGE_BYTES);
+
+ signs = 0;
+ for (i = 0; i < 8; ++i)
+ signs |= (uint64_t)buf[i] << 8 * i;
+ pos = 8;
+
+ for (i = 0; i < DILITHIUM_N; ++i)
+ c->coeffs[i] = 0;
+
+ for (i = DILITHIUM_N - DILITHIUM_TAU; i < DILITHIUM_N; ++i) {
+ do {
+ if (pos >= SHAKE256_BLOCK_SIZE) {
+ shake256_squeeze(&hash_ctx, buf, POLY_CHALLENGE_BYTES);
+ pos = 0;
+ }
+
+ b = buf[pos++];
+ } while (b > i);
+
+ c->coeffs[i] = c->coeffs[b];
+ c->coeffs[b] = 1 - (int32_t)(2 * (signs & 1));
+ signs >>= 1;
+ }
+
+ shake256_clear(&hash_ctx);
+}
+
+/**
+ * @brief polyeta_pack - Bit-pack polynomial with coefficients in [-ETA,ETA].
+ *
+ * @param [out] r pointer to output byte array with at least
+ * DILITHIUM_POLYETA_PACKEDBYTES bytes
+ * @param [in] a pointer to input polynomial
+ */
+void polyeta_pack(uint8_t *r, const poly *a)
+{
+ unsigned int i;
+ uint8_t t[8];
+
+#if DILITHIUM_ETA == 2
+ for (i = 0; i < DILITHIUM_N / 8; ++i) {
+ t[0] = (uint8_t)(DILITHIUM_ETA - a->coeffs[8 * i + 0]);
+ t[1] = (uint8_t)(DILITHIUM_ETA - a->coeffs[8 * i + 1]);
+ t[2] = (uint8_t)(DILITHIUM_ETA - a->coeffs[8 * i + 2]);
+ t[3] = (uint8_t)(DILITHIUM_ETA - a->coeffs[8 * i + 3]);
+ t[4] = (uint8_t)(DILITHIUM_ETA - a->coeffs[8 * i + 4]);
+ t[5] = (uint8_t)(DILITHIUM_ETA - a->coeffs[8 * i + 5]);
+ t[6] = (uint8_t)(DILITHIUM_ETA - a->coeffs[8 * i + 6]);
+ t[7] = (uint8_t)(DILITHIUM_ETA - a->coeffs[8 * i + 7]);
+
+ r[3 * i + 0] =
+ (uint8_t)((t[0] >> 0) | (t[1] << 3) | (t[2] << 6));
+ r[3 * i + 1] = (uint8_t)((t[2] >> 2) | (t[3] << 1) |
+ (t[4] << 4) | (t[5] << 7));
+ r[3 * i + 2] =
+ (uint8_t)((t[5] >> 1) | (t[6] << 2) | (t[7] << 5));
+ }
+#elif DILITHIUM_ETA == 4
+ for (i = 0; i < DILITHIUM_N / 2; ++i) {
+ t[0] = (uint8_t)(DILITHIUM_ETA - a->coeffs[2 * i + 0]);
+ t[1] = (uint8_t)(DILITHIUM_ETA - a->coeffs[2 * i + 1]);
+ r[i] = (uint8_t)(t[0] | (t[1] << 4));
+ }
+#else
+#error "Undefined DILITHIUM_ETA"
+#endif
+}
+
+/**
+ * @brief polyeta_unpack - Unpack polynomial with coefficients in [-ETA,ETA].
+ *
+ * @param [out] r pointer to output polynomial
+ * @param [in] a byte array with bit-packed polynomial
+ */
+void polyeta_unpack(poly *r, const uint8_t *a)
+{
+ unsigned int i;
+
+#if DILITHIUM_ETA == 2
+ for (i = 0; i < DILITHIUM_N / 8; ++i) {
+ r->coeffs[8 * i + 0] = (a[3 * i + 0] >> 0) & 7;
+ r->coeffs[8 * i + 1] = (a[3 * i + 0] >> 3) & 7;
+ r->coeffs[8 * i + 2] =
+ ((a[3 * i + 0] >> 6) | (a[3 * i + 1] << 2)) & 7;
+ r->coeffs[8 * i + 3] = (a[3 * i + 1] >> 1) & 7;
+ r->coeffs[8 * i + 4] = (a[3 * i + 1] >> 4) & 7;
+ r->coeffs[8 * i + 5] =
+ ((a[3 * i + 1] >> 7) | (a[3 * i + 2] << 1)) & 7;
+ r->coeffs[8 * i + 6] = (a[3 * i + 2] >> 2) & 7;
+ r->coeffs[8 * i + 7] = (a[3 * i + 2] >> 5) & 7;
+
+ r->coeffs[8 * i + 0] = DILITHIUM_ETA - r->coeffs[8 * i + 0];
+ r->coeffs[8 * i + 1] = DILITHIUM_ETA - r->coeffs[8 * i + 1];
+ r->coeffs[8 * i + 2] = DILITHIUM_ETA - r->coeffs[8 * i + 2];
+ r->coeffs[8 * i + 3] = DILITHIUM_ETA - r->coeffs[8 * i + 3];
+ r->coeffs[8 * i + 4] = DILITHIUM_ETA - r->coeffs[8 * i + 4];
+ r->coeffs[8 * i + 5] = DILITHIUM_ETA - r->coeffs[8 * i + 5];
+ r->coeffs[8 * i + 6] = DILITHIUM_ETA - r->coeffs[8 * i + 6];
+ r->coeffs[8 * i + 7] = DILITHIUM_ETA - r->coeffs[8 * i + 7];
+ }
+#elif DILITHIUM_ETA == 4
+ for (i = 0; i < DILITHIUM_N / 2; ++i) {
+ r->coeffs[2 * i + 0] = a[i] & 0x0F;
+ r->coeffs[2 * i + 1] = a[i] >> 4;
+ r->coeffs[2 * i + 0] = DILITHIUM_ETA - r->coeffs[2 * i + 0];
+ r->coeffs[2 * i + 1] = DILITHIUM_ETA - r->coeffs[2 * i + 1];
+ }
+#else
+#error "Undefined DILITHIUM_ETA"
+#endif
+}
+
+/**
+ * @brief polyt1_pack - Bit-pack polynomial t1 with coefficients fitting in 10
+ * bits. Input coefficients are assumed to be standard
+ * representatives.
+ *
+ * @param [out] r pointer to output byte array with at least
+ * DILITHIUM_POLYT1_PACKEDBYTES bytes
+ * @param [in] a pointer to input polynomial
+ */
+void polyt1_pack(uint8_t *r, const poly *a)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_N / 4; ++i) {
+ r[5 * i + 0] = (uint8_t)((a->coeffs[4 * i + 0] >> 0));
+ r[5 * i + 1] = (uint8_t)((a->coeffs[4 * i + 0] >> 8) |
+ (a->coeffs[4 * i + 1] << 2));
+ r[5 * i + 2] = (uint8_t)((a->coeffs[4 * i + 1] >> 6) |
+ (a->coeffs[4 * i + 2] << 4));
+ r[5 * i + 3] = (uint8_t)((a->coeffs[4 * i + 2] >> 4) |
+ (a->coeffs[4 * i + 3] << 6));
+ r[5 * i + 4] = (uint8_t)((a->coeffs[4 * i + 3] >> 2));
+ }
+}
+
+/**
+ * @brief polyt0_pack - Bit-pack polynomial t0 with coefficients in
+ * ]-2^{D-1}, 2^{D-1}].
+ *
+ * @param [out] r pointer to output byte array with at least
+ * DILITHIUM_POLYT0_PACKEDBYTES bytes
+ * @param [in] a pointer to input polynomial
+ */
+void polyt0_pack(uint8_t *r, const poly *a)
+{
+ unsigned int i;
+ uint32_t t[8];
+
+ for (i = 0; i < DILITHIUM_N / 8; ++i) {
+ t[0] = (uint32_t)((1 << (DILITHIUM_D - 1)) -
+ a->coeffs[8 * i + 0]);
+ t[1] = (uint32_t)((1 << (DILITHIUM_D - 1)) -
+ a->coeffs[8 * i + 1]);
+ t[2] = (uint32_t)((1 << (DILITHIUM_D - 1)) -
+ a->coeffs[8 * i + 2]);
+ t[3] = (uint32_t)((1 << (DILITHIUM_D - 1)) -
+ a->coeffs[8 * i + 3]);
+ t[4] = (uint32_t)((1 << (DILITHIUM_D - 1)) -
+ a->coeffs[8 * i + 4]);
+ t[5] = (uint32_t)((1 << (DILITHIUM_D - 1)) -
+ a->coeffs[8 * i + 5]);
+ t[6] = (uint32_t)((1 << (DILITHIUM_D - 1)) -
+ a->coeffs[8 * i + 6]);
+ t[7] = (uint32_t)((1 << (DILITHIUM_D - 1)) -
+ a->coeffs[8 * i + 7]);
+
+ r[13 * i + 0] = (uint8_t)(t[0]);
+ r[13 * i + 1] = (uint8_t)(t[0] >> 8);
+ r[13 * i + 1] |= (uint8_t)(t[1] << 5);
+ r[13 * i + 2] = (uint8_t)(t[1] >> 3);
+ r[13 * i + 3] = (uint8_t)(t[1] >> 11);
+ r[13 * i + 3] |= (uint8_t)(t[2] << 2);
+ r[13 * i + 4] = (uint8_t)(t[2] >> 6);
+ r[13 * i + 4] |= (uint8_t)(t[3] << 7);
+ r[13 * i + 5] = (uint8_t)(t[3] >> 1);
+ r[13 * i + 6] = (uint8_t)(t[3] >> 9);
+ r[13 * i + 6] |= (uint8_t)(t[4] << 4);
+ r[13 * i + 7] = (uint8_t)(t[4] >> 4);
+ r[13 * i + 8] = (uint8_t)(t[4] >> 12);
+ r[13 * i + 8] |= (uint8_t)(t[5] << 1);
+ r[13 * i + 9] = (uint8_t)(t[5] >> 7);
+ r[13 * i + 9] |= (uint8_t)(t[6] << 6);
+ r[13 * i + 10] = (uint8_t)(t[6] >> 2);
+ r[13 * i + 11] = (uint8_t)(t[6] >> 10);
+ r[13 * i + 11] |= (uint8_t)(t[7] << 3);
+ r[13 * i + 12] = (uint8_t)(t[7] >> 5);
+ }
+
+ memzero_explicit(t, sizeof(t));
+}
+
+/**
+ * @brief polyt0_unpack - Unpack polynomial t0 with coefficients in
+ * ]-2^{D-1}, 2^{D-1}].
+ *
+ * @param [out] r pointer to output polynomial
+ * @param [in] a byte array with bit-packed polynomial
+ */
+void polyt0_unpack(poly *r, const uint8_t *a)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_N / 8; ++i) {
+ r->coeffs[8 * i + 0] = a[13 * i + 0];
+ r->coeffs[8 * i + 0] |= (int32_t)a[13 * i + 1] << 8;
+ r->coeffs[8 * i + 0] &= 0x1FFF;
+
+ r->coeffs[8 * i + 1] = a[13 * i + 1] >> 5;
+ r->coeffs[8 * i + 1] |= (int32_t)a[13 * i + 2] << 3;
+ r->coeffs[8 * i + 1] |= (int32_t)a[13 * i + 3] << 11;
+ r->coeffs[8 * i + 1] &= 0x1FFF;
+
+ r->coeffs[8 * i + 2] = a[13 * i + 3] >> 2;
+ r->coeffs[8 * i + 2] |= (int32_t)a[13 * i + 4] << 6;
+ r->coeffs[8 * i + 2] &= 0x1FFF;
+
+ r->coeffs[8 * i + 3] = a[13 * i + 4] >> 7;
+ r->coeffs[8 * i + 3] |= (int32_t)a[13 * i + 5] << 1;
+ r->coeffs[8 * i + 3] |= (int32_t)a[13 * i + 6] << 9;
+ r->coeffs[8 * i + 3] &= 0x1FFF;
+
+ r->coeffs[8 * i + 4] = a[13 * i + 6] >> 4;
+ r->coeffs[8 * i + 4] |= (int32_t)a[13 * i + 7] << 4;
+ r->coeffs[8 * i + 4] |= (int32_t)a[13 * i + 8] << 12;
+ r->coeffs[8 * i + 4] &= 0x1FFF;
+
+ r->coeffs[8 * i + 5] = a[13 * i + 8] >> 1;
+ r->coeffs[8 * i + 5] |= (int32_t)a[13 * i + 9] << 7;
+ r->coeffs[8 * i + 5] &= 0x1FFF;
+
+ r->coeffs[8 * i + 6] = a[13 * i + 9] >> 6;
+ r->coeffs[8 * i + 6] |= (int32_t)a[13 * i + 10] << 2;
+ r->coeffs[8 * i + 6] |= (int32_t)a[13 * i + 11] << 10;
+ r->coeffs[8 * i + 6] &= 0x1FFF;
+
+ r->coeffs[8 * i + 7] = a[13 * i + 11] >> 3;
+ r->coeffs[8 * i + 7] |= (int32_t)a[13 * i + 12] << 5;
+ r->coeffs[8 * i + 7] &= 0x1FFF;
+
+ r->coeffs[8 * i + 0] =
+ (1 << (DILITHIUM_D - 1)) - r->coeffs[8 * i + 0];
+ r->coeffs[8 * i + 1] =
+ (1 << (DILITHIUM_D - 1)) - r->coeffs[8 * i + 1];
+ r->coeffs[8 * i + 2] =
+ (1 << (DILITHIUM_D - 1)) - r->coeffs[8 * i + 2];
+ r->coeffs[8 * i + 3] =
+ (1 << (DILITHIUM_D - 1)) - r->coeffs[8 * i + 3];
+ r->coeffs[8 * i + 4] =
+ (1 << (DILITHIUM_D - 1)) - r->coeffs[8 * i + 4];
+ r->coeffs[8 * i + 5] =
+ (1 << (DILITHIUM_D - 1)) - r->coeffs[8 * i + 5];
+ r->coeffs[8 * i + 6] =
+ (1 << (DILITHIUM_D - 1)) - r->coeffs[8 * i + 6];
+ r->coeffs[8 * i + 7] =
+ (1 << (DILITHIUM_D - 1)) - r->coeffs[8 * i + 7];
+ }
+}
+
+/**
+ * @param polyz_pack - Bit-pack polynomial with coefficients
+ * in [-(GAMMA1 - 1), GAMMA1].
+ *
+ * @param [out] r pointer to output byte array with at least
+ * DILITHIUM_POLYZ_PACKEDBYTES bytes
+ * @param [in] a pointer to input polynomial
+ */
+void polyz_pack(uint8_t *r, const poly *a)
+{
+ unsigned int i;
+ uint32_t t[4];
+
+#if DILITHIUM_GAMMA1 == (1 << 17)
+ for (i = 0; i < DILITHIUM_N / 4; ++i) {
+ t[0] = (uint32_t)(DILITHIUM_GAMMA1 - a->coeffs[4 * i + 0]);
+ t[1] = (uint32_t)(DILITHIUM_GAMMA1 - a->coeffs[4 * i + 1]);
+ t[2] = (uint32_t)(DILITHIUM_GAMMA1 - a->coeffs[4 * i + 2]);
+ t[3] = (uint32_t)(DILITHIUM_GAMMA1 - a->coeffs[4 * i + 3]);
+
+ r[9 * i + 0] = (uint8_t)(t[0]);
+ r[9 * i + 1] = (uint8_t)(t[0] >> 8);
+ r[9 * i + 2] = (uint8_t)(t[0] >> 16);
+ r[9 * i + 2] |= (uint8_t)(t[1] << 2);
+ r[9 * i + 3] = (uint8_t)(t[1] >> 6);
+ r[9 * i + 4] = (uint8_t)(t[1] >> 14);
+ r[9 * i + 4] |= (uint8_t)(t[2] << 4);
+ r[9 * i + 5] = (uint8_t)(t[2] >> 4);
+ r[9 * i + 6] = (uint8_t)(t[2] >> 12);
+ r[9 * i + 6] |= (uint8_t)(t[3] << 6);
+ r[9 * i + 7] = (uint8_t)(t[3] >> 2);
+ r[9 * i + 8] = (uint8_t)(t[3] >> 10);
+ }
+#elif DILITHIUM_GAMMA1 == (1 << 19)
+ for (i = 0; i < DILITHIUM_N / 2; ++i) {
+ t[0] = (uint32_t)(DILITHIUM_GAMMA1 - a->coeffs[2 * i + 0]);
+ t[1] = (uint32_t)(DILITHIUM_GAMMA1 - a->coeffs[2 * i + 1]);
+
+ r[5 * i + 0] = (uint8_t)(t[0]);
+ r[5 * i + 1] = (uint8_t)(t[0] >> 8);
+ r[5 * i + 2] = (uint8_t)(t[0] >> 16);
+ r[5 * i + 2] |= (uint8_t)(t[1] << 4);
+ r[5 * i + 3] = (uint8_t)(t[1] >> 4);
+ r[5 * i + 4] = (uint8_t)(t[1] >> 12);
+ }
+#else
+#error "Undefined Gamma"
+#endif
+
+ memzero_explicit(t, sizeof(t));
+}
+
+/**
+ * @brief polyz_unpack - Unpack polynomial z with coefficients
+ * in [-(GAMMA1 - 1), GAMMA1].
+ *
+ * @param [out] r pointer to output polynomial
+ * @param [in] a byte array with bit-packed polynomial
+ */
+void polyz_unpack(poly *r, const uint8_t *a)
+{
+ unsigned int i;
+
+#if DILITHIUM_GAMMA1 == (1 << 17)
+ for (i = 0; i < DILITHIUM_N / 4; ++i) {
+ r->coeffs[4 * i + 0] = a[9 * i + 0];
+ r->coeffs[4 * i + 0] |= (int32_t)a[9 * i + 1] << 8;
+ r->coeffs[4 * i + 0] |= (int32_t)a[9 * i + 2] << 16;
+ r->coeffs[4 * i + 0] &= 0x3FFFF;
+
+ r->coeffs[4 * i + 1] = a[9 * i + 2] >> 2;
+ r->coeffs[4 * i + 1] |= (int32_t)a[9 * i + 3] << 6;
+ r->coeffs[4 * i + 1] |= (int32_t)a[9 * i + 4] << 14;
+ r->coeffs[4 * i + 1] &= 0x3FFFF;
+
+ r->coeffs[4 * i + 2] = a[9 * i + 4] >> 4;
+ r->coeffs[4 * i + 2] |= (int32_t)a[9 * i + 5] << 4;
+ r->coeffs[4 * i + 2] |= (int32_t)a[9 * i + 6] << 12;
+ r->coeffs[4 * i + 2] &= 0x3FFFF;
+
+ r->coeffs[4 * i + 3] = a[9 * i + 6] >> 6;
+ r->coeffs[4 * i + 3] |= (int32_t)a[9 * i + 7] << 2;
+ r->coeffs[4 * i + 3] |= (int32_t)a[9 * i + 8] << 10;
+ r->coeffs[4 * i + 3] &= 0x3FFFF;
+
+ r->coeffs[4 * i + 0] =
+ DILITHIUM_GAMMA1 - r->coeffs[4 * i + 0];
+ r->coeffs[4 * i + 1] =
+ DILITHIUM_GAMMA1 - r->coeffs[4 * i + 1];
+ r->coeffs[4 * i + 2] =
+ DILITHIUM_GAMMA1 - r->coeffs[4 * i + 2];
+ r->coeffs[4 * i + 3] =
+ DILITHIUM_GAMMA1 - r->coeffs[4 * i + 3];
+ }
+#elif DILITHIUM_GAMMA1 == (1 << 19)
+ for (i = 0; i < DILITHIUM_N / 2; ++i) {
+ r->coeffs[2 * i + 0] = a[5 * i + 0];
+ r->coeffs[2 * i + 0] |= (int32_t)a[5 * i + 1] << 8;
+ r->coeffs[2 * i + 0] |= (int32_t)a[5 * i + 2] << 16;
+ r->coeffs[2 * i + 0] &= 0xFFFFF;
+
+ r->coeffs[2 * i + 1] = a[5 * i + 2] >> 4;
+ r->coeffs[2 * i + 1] |= (int32_t)a[5 * i + 3] << 4;
+ r->coeffs[2 * i + 1] |= (int32_t)a[5 * i + 4] << 12;
+ r->coeffs[2 * i + 1] &= 0xFFFFF;
+
+ r->coeffs[2 * i + 0] =
+ DILITHIUM_GAMMA1 - r->coeffs[2 * i + 0];
+ r->coeffs[2 * i + 1] =
+ DILITHIUM_GAMMA1 - r->coeffs[2 * i + 1];
+ }
+#else
+#error "Undefined Gamma"
+#endif
+}
+
+/**
+ * @brief polyw1_pack - Bit-pack polynomial w1 with coefficients in [0,15] or
+ * [0,43]. Input coefficients are assumed to be standard
+ * representatives.
+ *
+ * @param [out] r pointer to output byte array with at least
+ * DILITHIUM_POLYW1_PACKEDBYTES bytes
+ * @param [in] a pointer to input polynomial
+ */
+void polyw1_pack(uint8_t *r, const poly *a)
+{
+ unsigned int i;
+
+#if DILITHIUM_GAMMA2 == (DILITHIUM_Q - 1) / 88
+ for (i = 0; i < DILITHIUM_N / 4; ++i) {
+ r[3 * i + 0] = (uint8_t)(a->coeffs[4 * i + 0]);
+ r[3 * i + 0] |= (uint8_t)(a->coeffs[4 * i + 1] << 6);
+ r[3 * i + 1] = (uint8_t)(a->coeffs[4 * i + 1] >> 2);
+ r[3 * i + 1] |= (uint8_t)(a->coeffs[4 * i + 2] << 4);
+ r[3 * i + 2] = (uint8_t)(a->coeffs[4 * i + 2] >> 4);
+ r[3 * i + 2] |= (uint8_t)(a->coeffs[4 * i + 3] << 2);
+ }
+#elif DILITHIUM_GAMMA2 == (DILITHIUM_Q - 1) / 32
+ for (i = 0; i < DILITHIUM_N / 2; ++i)
+ r[i] = (uint8_t)(a->coeffs[2 * i + 0] |
+ (a->coeffs[2 * i + 1] << 4));
+#else
+#error "Undefined Gamma"
+#endif
+}
diff --git a/crypto/ml_dsa/dilithium_poly.h b/crypto/ml_dsa/dilithium_poly.h
new file mode 100644
index 000000000000..a894f5483fe8
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_poly.h
@@ -0,0 +1,190 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#ifndef DILITHIUM_POLY_H
+#define DILITHIUM_POLY_H
+
+#include "dilithium_type.h"
+#include "dilithium_reduce.h"
+#include "dilithium_rounding.h"
+
+typedef struct {
+ int32_t coeffs[DILITHIUM_N];
+} poly;
+
+/**
+ * @brief poly_add - Add polynomials. No modular reduction is performed.
+ *
+ * @param [out] c pointer to output polynomial
+ * @param [in] a pointer to first summand
+ * @param [in] b pointer to second summand
+ */
+static inline void poly_add(poly *c, const poly *a, const poly *b)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_N; ++i)
+ c->coeffs[i] = a->coeffs[i] + b->coeffs[i];
+}
+
+/**
+ * @brief poly_sub - Subtract polynomials. No modular reduction is
+ * performed.
+ *
+ * @param [out] c pointer to output polynomial
+ * @param [in] a pointer to first input polynomial
+ * @param [in] b pointer to second input polynomial to be subtraced from first
+ * input polynomial
+ */
+static inline void poly_sub(poly *c, const poly *a, const poly *b)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_N; ++i)
+ c->coeffs[i] = a->coeffs[i] - b->coeffs[i];
+}
+
+/**
+ * @brief poly_shiftl - Multiply polynomial by 2^D without modular reduction.
+ * Assumes input coefficients to be less than 2^{31-D} in
+ * absolute value.
+ *
+ * @param [in,out] a pointer to input/output polynomial
+ */
+static inline void poly_shiftl(poly *a)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_N; ++i)
+ a->coeffs[i] <<= DILITHIUM_D;
+}
+
+/**
+ * @brief poly_decompose - For all coefficients c of the input polynomial,
+ * compute high and low bits c0, c1 such
+ * c mod Q = c1*ALPHA + c0 with
+ * -ALPHA/2 < c0 <= ALPHA/2 except c1 = (Q-1)/ALPHA
+ * where we set c1 = 0 and
+ * -ALPHA/2 <= c0 = c mod Q - Q < 0.
+ * Assumes coefficients to be standard representatives.
+ *
+ * @param [out] a1 pointer to output polynomial with coefficients c1
+ * @param [out] a0 pointer to output polynomial with coefficients c0
+ * @param [in] a pointer to input polynomial
+ */
+static inline void poly_decompose(poly *a1, poly *a0, const poly *a)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_N; ++i)
+ a1->coeffs[i] = decompose(&a0->coeffs[i], a->coeffs[i]);
+}
+
+/**
+ * @brief poly_make_hint - Compute hint polynomial. The coefficients of which
+ * indicate whether the low bits of the corresponding
+ * coefficient of the input polynomial overflow into the
+ * high bits.
+ *
+ * @param [out] h pointer to output hint polynomial
+ * @param [in] a0 pointer to low part of input polynomial
+ * @param [in] a1 pointer to high part of input polynomial
+ *
+ * @return number of 1 bits.
+ */
+static inline unsigned int poly_make_hint(poly *h, const poly *a0,
+ const poly *a1)
+{
+ unsigned int i, s = 0;
+
+ for (i = 0; i < DILITHIUM_N; ++i) {
+ h->coeffs[i] = make_hint(a0->coeffs[i], a1->coeffs[i]);
+ s += (unsigned int)h->coeffs[i];
+ }
+
+ return s;
+}
+
+/**
+ * @brief poly_use_hint - Use hint polynomial to correct the high bits of a
+ * polynomial.
+ *
+ * @param [out] b pointer to output polynomial with corrected high bits
+ * @param [in] a pointer to input polynomial
+ * @param [in] h pointer to input hint polynomial
+ */
+static inline void poly_use_hint(poly *b, const poly *a, const poly *h)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_N; ++i)
+ b->coeffs[i] = use_hint(a->coeffs[i], h->coeffs[i]);
+}
+
+int poly_chknorm(const poly *a, int32_t B);
+
+#define POLY_UNIFORM_NBLOCKS \
+ ((768 + SHAKE128_BLOCK_SIZE - 1) / SHAKE128_BLOCK_SIZE)
+void poly_uniform(poly *a, const uint8_t seed[DILITHIUM_SEEDBYTES],
+ uint16_t nonce, void *ws_buf);
+
+#if DILITHIUM_ETA == 2
+#define POLY_UNIFORM_ETA_NBLOCKS \
+ ((136 + SHAKE256_BLOCK_SIZE - 1) / SHAKE256_BLOCK_SIZE)
+#elif DILITHIUM_ETA == 4
+#define POLY_UNIFORM_ETA_NBLOCKS \
+ ((227 + SHAKE256_BLOCK_SIZE - 1) / SHAKE256_BLOCK_SIZE)
+#else
+#error "Undefined DILITHIUM_ETA"
+#endif
+#define POLY_UNIFORM_ETA_BYTES POLY_UNIFORM_ETA_NBLOCKS *SHAKE256_BLOCK_SIZE
+void poly_uniform_eta(poly *a, const uint8_t seed[DILITHIUM_CRHBYTES],
+ uint16_t nonce, void *ws_buf);
+
+#define POLY_UNIFORM_GAMMA1_NBLOCKS \
+ ((DILITHIUM_POLYZ_PACKEDBYTES + SHAKE256_BLOCK_SIZE - 1) / \
+ SHAKE256_BLOCK_SIZE)
+#define POLY_UNIFORM_GAMMA1_BYTES \
+ POLY_UNIFORM_GAMMA1_NBLOCKS *SHAKE256_BLOCK_SIZE
+
+#define POLY_CHALLENGE_BYTES SHAKE256_BLOCK_SIZE
+void poly_challenge(poly *c, const uint8_t seed[DILITHIUM_CTILDE_BYTES],
+ void *ws_buf);
+
+void polyeta_pack(uint8_t *r, const poly *a);
+void polyeta_unpack(poly *r, const uint8_t *a);
+
+void polyt1_pack(uint8_t *r, const poly *a);
+
+void polyt0_pack(uint8_t *r, const poly *a);
+void polyt0_unpack(poly *r, const uint8_t *a);
+
+void polyz_pack(uint8_t *r, const poly *a);
+void polyz_unpack(poly *r, const uint8_t *a);
+
+void polyw1_pack(uint8_t *r, const poly *a);
+
+#endif /* DILITHIUM_POLY_H */
diff --git a/crypto/ml_dsa/dilithium_poly_c.h b/crypto/ml_dsa/dilithium_poly_c.h
new file mode 100644
index 000000000000..ca0107f75bab
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_poly_c.h
@@ -0,0 +1,149 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2023 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+
+#ifndef DILITHIUM_POLY_C_H
+#define DILITHIUM_POLY_C_H
+
+#include "dilithium_ntt.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * @brief poly_reduce - Inplace reduction of all coefficients of polynomial to
+ * representative in [-6283009,6283007].
+ *
+ * @param [in,out] a pointer to input/output polynomial
+ */
+static inline void poly_reduce(poly *a)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_N; ++i)
+ a->coeffs[i] = reduce32(a->coeffs[i]);
+}
+
+/**
+ * @brief poly_caddq - For all coefficients of in/out polynomial add Q if
+ * coefficient is negative.
+ *
+ * @param [in,out] a pointer to input/output polynomial
+ */
+static inline void poly_caddq(poly *a)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_N; ++i)
+ a->coeffs[i] = caddq(a->coeffs[i]);
+}
+
+/**
+ * @brief poly_pointwise_montgomery - Pointwise multiplication of polynomials in
+ * NTT domain representation and
+ * multiplication of resulting polynomial
+ * by 2^{-32}.
+ *
+ * @param [out] c pointer to output polynomial
+ * @param [in] a pointer to first input polynomial
+ * @param [in] b pointer to second input polynomial
+ */
+static inline void poly_pointwise_montgomery(poly *c, const poly *a,
+ const poly *b)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_N; ++i)
+ c->coeffs[i] =
+ montgomery_reduce((int64_t)a->coeffs[i] * b->coeffs[i]);
+}
+
+/**
+ * @brief poly_power2round - For all coefficients c of the input polynomial,
+ * compute c0, c1 such that c mod Q = c1*2^D + c0
+ * with -2^{D-1} < c0 <= 2^{D-1}. Assumes coefficients
+ * to be standard representatives.
+ *
+ * @param [out] a1 pointer to output polynomial with coefficients c1
+ * @param [out] a0 pointer to output polynomial with coefficients c0
+ * @param [in] a pointer to input polynomial
+ */
+static inline void poly_power2round(poly *a1, poly *a0, const poly *a)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_N; ++i)
+ a1->coeffs[i] = power2round(&a0->coeffs[i], a->coeffs[i]);
+}
+
+/**
+ * @brief polyt1_unpack - Unpack polynomial t1 with 10-bit coefficients.
+ * Output coefficients are standard representatives.
+ *
+ * @param [out] r pointer to output polynomial
+ * @param [in] a byte array with bit-packed polynomial
+ */
+static inline void polyt1_unpack(poly *r, const uint8_t *a)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_N / 4; ++i) {
+ r->coeffs[4 * i + 0] =
+ ((a[5 * i + 0] >> 0) | ((uint32_t)a[5 * i + 1] << 8)) &
+ 0x3FF;
+ r->coeffs[4 * i + 1] =
+ ((a[5 * i + 1] >> 2) | ((uint32_t)a[5 * i + 2] << 6)) &
+ 0x3FF;
+ r->coeffs[4 * i + 2] =
+ ((a[5 * i + 2] >> 4) | ((uint32_t)a[5 * i + 3] << 4)) &
+ 0x3FF;
+ r->coeffs[4 * i + 3] =
+ ((a[5 * i + 3] >> 6) | ((uint32_t)a[5 * i + 4] << 2)) &
+ 0x3FF;
+ }
+}
+
+/**
+ * @brief poly_ntt - Inplace forward NTT. Coefficients can grow by
+ * 8*Q in absolute value.
+ *
+ * @param [in,out] a pointer to input/output polynomial
+ */
+static inline void poly_ntt(poly *a)
+{
+ ntt(a->coeffs);
+}
+
+/**
+ * @brief poly_invntt_tomont - Inplace inverse NTT and multiplication by 2^{32}.
+ * Input coefficients need to be less than Q in
+ * absolute value and output coefficients are again
+ * bounded by Q.
+ *
+ * @param [in,out] a pointer to input/output polynomial
+ */
+static inline void poly_invntt_tomont(poly *a)
+{
+ invntt_tomont(a->coeffs);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* DILITHIUM_POLY_C_H */
diff --git a/crypto/ml_dsa/dilithium_poly_common.h b/crypto/ml_dsa/dilithium_poly_common.h
new file mode 100644
index 000000000000..7c7cef28b545
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_poly_common.h
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#ifndef DILITHIUM_POLY_COMMON_H
+#define DILITHIUM_POLY_COMMON_H
+
+#include "dilithium_type.h"
+
+void poly_uniform_gamma1(poly *a, const uint8_t seed[DILITHIUM_CRHBYTES],
+ uint16_t nonce, void *ws_buf);
+
+#endif /* DILITHIUM_POLY_COMMON_H */
diff --git a/crypto/ml_dsa/dilithium_polyvec.h b/crypto/ml_dsa/dilithium_polyvec.h
new file mode 100644
index 000000000000..7e428e3becfd
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_polyvec.h
@@ -0,0 +1,343 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#ifndef DILITHIUM_POLYVEC_H
+#define DILITHIUM_POLYVEC_H
+
+#include "dilithium_type.h"
+#include "dilithium_poly.h"
+
+typedef struct {
+ poly vec[DILITHIUM_L];
+} polyvecl;
+
+/* Vectors of polynomials of length K */
+typedef struct {
+ poly vec[DILITHIUM_K];
+} polyveck;
+
+/**************************************************************/
+/************ Vectors of polynomials of length L **************/
+/**************************************************************/
+
+static inline void polyvecl_reduce(polyvecl *v)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_L; ++i)
+ poly_reduce(&v->vec[i]);
+}
+
+/**
+ * @brief polyvecl_add - Add vectors of polynomials of length L.
+ * No modular reduction is performed.
+ *
+ * @param [out] w pointer to output vector
+ * @param [in] u pointer to first summand
+ * @param [in] v pointer to second summand
+ */
+static inline void polyvecl_add(polyvecl *w, const polyvecl *u,
+ const polyvecl *v)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_L; ++i)
+ poly_add(&w->vec[i], &u->vec[i], &v->vec[i]);
+}
+
+/**
+ * @brief polyvecl_ntt - Forward NTT of all polynomials in vector of length L.
+ * Output coefficients can be up to 16*Q larger than input
+ * coefficients.
+ *
+ * @param [in,out] v pointer to input/output vector
+ */
+static inline void polyvecl_ntt(polyvecl *v)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_L; ++i)
+ poly_ntt(&v->vec[i]);
+}
+
+static inline void polyvecl_invntt_tomont(polyvecl *v)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_L; ++i)
+ poly_invntt_tomont(&v->vec[i]);
+}
+
+static inline void polyvecl_pointwise_poly_montgomery(polyvecl *r,
+ const poly *a,
+ const polyvecl *v)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_L; ++i)
+ poly_pointwise_montgomery(&r->vec[i], a, &v->vec[i]);
+}
+
+/**
+ * @brief polyvecl_chknorm - Check infinity norm of polynomials in vector of
+ * length L. Assumes input polyvecl to be reduced by
+ * polyvecl_reduce().
+ *
+ * @param [in] v pointer to vector
+ * @param [in] bound norm bound
+ *
+ * @return 0 if norm of all polynomials is strictly smaller than B <= (Q-1)/8
+ * and 1 otherwise.
+ */
+static inline int polyvecl_chknorm(const polyvecl *v, int32_t bound)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_L; ++i)
+ if (poly_chknorm(&v->vec[i], bound))
+ return 1;
+
+ return 0;
+}
+
+/**************************************************************/
+/************ Vectors of polynomials of length K **************/
+/**************************************************************/
+
+/**
+ * @brief polyveck_reduce - Reduce coefficients of polynomials in vector of
+ * length DILITHIUM_K to representatives in
+ * [-6283009,6283007].
+ *
+ * @param [in,out] v pointer to input/output vector
+ */
+static inline void polyveck_reduce(polyveck *v)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ poly_reduce(&v->vec[i]);
+}
+
+/**
+ * @brief polyveck_caddq - For all coefficients of polynomials in vector of
+ * length DILITHIUM_K add DILITHIUM_Q if
+ * coefficient is negative.
+ *
+ * @param [in,out] v pointer to input/output vector
+ */
+static inline void polyveck_caddq(polyveck *v)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ poly_caddq(&v->vec[i]);
+}
+
+/**
+ * @brief polyveck_add - Add vectors of polynomials of length DILITHIUM_K.
+ * No modular reduction is performed.
+ *
+ * @param [out] w pointer to output vector
+ * @param [in] u pointer to first summand
+ * @param [in] v pointer to second summand
+ */
+static inline void polyveck_add(polyveck *w, const polyveck *u,
+ const polyveck *v)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ poly_add(&w->vec[i], &u->vec[i], &v->vec[i]);
+}
+
+/**
+ * @brief olyveck_sub - Subtract vectors of polynomials of length
+ * DILITHIUM_K. No modular reduction is performed.
+ *
+ * @param [out] w pointer to output vector
+ * @param [in] u pointer to first input vector
+ * @param [in] v pointer to second input vector to be subtracted from first
+ * input vector
+ */
+static inline void polyveck_sub(polyveck *w, const polyveck *u,
+ const polyveck *v)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ poly_sub(&w->vec[i], &u->vec[i], &v->vec[i]);
+}
+
+/**
+ * @brief polyveck_shiftl - Multiply vector of polynomials of Length K by
+ * 2^D without modular reduction. Assumes input
+ * coefficients to be less than 2^{31-D}.
+ *
+ * @param [in,out] v pointer to input/output vector
+ */
+static inline void polyveck_shiftl(polyveck *v)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ poly_shiftl(&v->vec[i]);
+}
+
+/**
+ * @brief polyveck_ntt - Forward NTT of all polynomials in vector of length K.
+ * Output coefficients can be up to 16*Q larger than input
+ * coefficients.
+ *
+ * @param [in,out] v pointer to input/output vector
+ */
+static inline void polyveck_ntt(polyveck *v)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ poly_ntt(&v->vec[i]);
+}
+
+/**
+ * @brief polyveck_invntt_tomont - Inverse NTT and multiplication by 2^{32} of
+ * polynomials in vector of length K. Input
+ * coefficients need to be less than 2*Q.
+ *
+ * @param [in,out] v pointer to input/output vector
+ */
+static inline void polyveck_invntt_tomont(polyveck *v)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ poly_invntt_tomont(&v->vec[i]);
+}
+
+static inline void polyveck_pointwise_poly_montgomery(polyveck *r,
+ const poly *a,
+ const polyveck *v)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ poly_pointwise_montgomery(&r->vec[i], a, &v->vec[i]);
+}
+
+/**
+ * @brief polyveck_chknorm - Check infinity norm of polynomials in vector of
+ * length K. Assumes input polyveck to be reduced by
+ * polyveck_reduce().
+ *
+ * @param [in] v pointer to vector
+ * @param [in] bound norm bound
+ *
+ * @return 0 if norm of all polynomials are strictly smaller than B <= (Q-1)/8
+ * and 1 otherwise.
+ */
+static inline int polyveck_chknorm(const polyveck *v, int32_t bound)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ if (poly_chknorm(&v->vec[i], bound))
+ return 1;
+
+ return 0;
+}
+
+/**
+ * @brief polyveck_decompose - For all coefficients a of polynomials in vector
+ * of length K, compute high and low bits a0, a1
+ * such a mod^+ Q = a1*ALPHA + a0 with
+ * -ALPHA/2 < a0 <= ALPHA/2 except a1 = (Q-1)/ALPHA
+ * where we set a1 = 0 and
+ * -ALPHA/2 <= a0 = a mod Q - Q < 0. Assumes
+ * coefficients to be standard representatives.
+ *
+ * @param [out] v1 pointer to output vector of polynomials with coefficients a1
+ * @param [in] v0 pointer to output vector of polynomials with coefficients a0
+ * @param [in] v pointer to input vector
+ */
+static inline void polyveck_decompose(polyveck *v1, polyveck *v0,
+ const polyveck *v)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ poly_decompose(&v1->vec[i], &v0->vec[i], &v->vec[i]);
+}
+
+/**
+ * @brief polyveck_make_hint - Compute hint vector.
+ *
+ * @param [out] h pointer to output vector
+ * @param [in] v0 pointer to low part of input vector
+ * @param [in] v1 pointer to high part of input vector
+ *
+ * @return number of 1 bits.
+ */
+static inline unsigned int polyveck_make_hint(polyveck *h, const polyveck *v0,
+ const polyveck *v1)
+{
+ unsigned int i, s = 0;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ s += poly_make_hint(&h->vec[i], &v0->vec[i], &v1->vec[i]);
+
+ return s;
+}
+
+/**
+ * @brief polyveck_use_hint - Use hint vector to correct the high bits of input
+ * vector.
+ *
+ * @param [out] w pointer to output vector of polynomials with corrected high
+ * bits
+ * @param [in] u pointer to input vector
+ * @param [in] h pointer to input hint vector
+ */
+static inline void polyveck_use_hint(polyveck *w, const polyveck *u,
+ const polyveck *h)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ poly_use_hint(&w->vec[i], &u->vec[i], &h->vec[i]);
+}
+
+static inline void
+polyveck_pack_w1(uint8_t r[DILITHIUM_K * DILITHIUM_POLYW1_PACKEDBYTES],
+ const polyveck *w1)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ polyw1_pack(&r[i * DILITHIUM_POLYW1_PACKEDBYTES],
+ &w1->vec[i]);
+}
+
+#endif /* DILITHIUM_POLYVEC_H */
diff --git a/crypto/ml_dsa/dilithium_polyvec_c.h b/crypto/ml_dsa/dilithium_polyvec_c.h
new file mode 100644
index 000000000000..0f8c2d914e4d
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_polyvec_c.h
@@ -0,0 +1,94 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2023 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+
+#ifndef DILITHIUM_POLYVEC_C_H
+#define DILITHIUM_POLYVEC_C_H
+
+static inline void
+polyvecl_uniform_gamma1(polyvecl *v, const uint8_t seed[DILITHIUM_CRHBYTES],
+ uint16_t nonce, void *ws_buf)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_L; ++i)
+ poly_uniform_gamma1(
+ &v->vec[i], seed,
+ cpu_to_le16((uint16_t)(DILITHIUM_L * nonce + i)),
+ ws_buf);
+}
+
+/**
+ * @brief expand_mat - Implementation of ExpandA. Generates matrix A with
+ * uniformly random coefficients a_{i,j} by performing
+ * rejection sampling on the output stream of
+ * SHAKE128(rho|j|i).
+ *
+ * @param [out] mat output matrix
+ * @param [in] rho byte array containing seed rho
+ */
+static inline void
+polyvec_matrix_expand(polyvecl mat[DILITHIUM_K],
+ const uint8_t rho[DILITHIUM_SEEDBYTES], void *ws_buf)
+{
+ unsigned int i, j;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ for (j = 0; j < DILITHIUM_L; ++j)
+ poly_uniform(
+ &mat[i].vec[j], rho,
+ cpu_to_le16((uint16_t)(i << 8) + (uint16_t)j),
+ ws_buf);
+}
+
+/**
+ * @brief polyvecl_pointwise_acc_montgomery -
+ * Pointwise multiply vectors of polynomials of length L, multiply
+ * resulting vector by 2^{-32} and add (accumulate) polynomials
+ * in it. Input/output vectors are in NTT domain representation.
+ *
+ * @param [out] w output polynomial
+ * @param [in] u pointer to first input vector
+ * @param [in] v pointer to second input vector
+ */
+static inline void polyvecl_pointwise_acc_montgomery(poly *w, const polyvecl *u,
+ const polyvecl *v,
+ void *ws_buf)
+{
+ unsigned int i;
+ poly *t = ws_buf;
+
+ poly_pointwise_montgomery(w, &u->vec[0], &v->vec[0]);
+ for (i = 1; i < DILITHIUM_L; ++i) {
+ poly_pointwise_montgomery(t, &u->vec[i], &v->vec[i]);
+ poly_add(w, w, t);
+ }
+}
+
+static inline void
+polyvec_matrix_pointwise_montgomery(polyveck *t,
+ const polyvecl mat[DILITHIUM_K],
+ const polyvecl *v, void *ws_buf)
+{
+ unsigned int i;
+
+ for (i = 0; i < DILITHIUM_K; ++i)
+ polyvecl_pointwise_acc_montgomery(&t->vec[i], &mat[i], v,
+ ws_buf);
+}
+
+#endif /* DILITHIUM_POLYVEC_C_H */
diff --git a/crypto/ml_dsa/dilithium_reduce.h b/crypto/ml_dsa/dilithium_reduce.h
new file mode 100644
index 000000000000..c4dd78be2575
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_reduce.h
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#ifndef DILITHIUM_REDUCE_H
+#define DILITHIUM_REDUCE_H
+
+#include "dilithium_type.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#define MONT -4186625 // 2^32 % Q
+#define QINV 58728449 // q^(-1) mod 2^32
+
+/**
+ * @brief montgomery_reduce - For finite field element a with
+ * -2^{31}Q <= a <= Q*2^31,
+ * compute r \equiv a*2^{-32} (mod Q) such that
+ * -Q < r < Q.
+ *
+ * @param [in] a finite field element
+ *
+ * @return r
+ */
+static inline int32_t montgomery_reduce(int64_t a)
+{
+ int32_t t;
+
+ t = (int32_t)a * QINV;
+ t = (int32_t)((a - (int64_t)t * DILITHIUM_Q) >> 32);
+ return t;
+}
+
+/**
+ * @brief reduce32 - For finite field element a with a <= 2^{31} - 2^{22} - 1,
+ * compute r \equiv a (mod Q) such that
+ * -6283009 <= r <= 6283007.
+ *
+ * @param [in] a finite field element
+ *
+ * @return r
+ */
+static inline int32_t reduce32(int32_t a)
+{
+ int32_t t;
+
+ t = (a + (1 << 22)) >> 23;
+ t = a - t * DILITHIUM_Q;
+ return t;
+}
+
+/**
+ * @brief caddq - Add Q if input coefficient is negative.
+ *
+ * @param [in] a finite field element
+ *
+ * @return r
+ */
+static inline int32_t caddq(int32_t a)
+{
+ a += (a >> 31) & DILITHIUM_Q;
+ return a;
+}
+
+/**
+ * @brief freeze - For finite field element a, compute standard representative
+ * r = a mod^+ Q.
+ *
+ * @param [in] a finite field element a
+ *
+ * @return r
+ */
+static inline int32_t freeze(int32_t a)
+{
+ a = reduce32(a);
+ a = caddq(a);
+ return a;
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* DILITHIUM_REDUCE_H */
diff --git a/crypto/ml_dsa/dilithium_rounding.c b/crypto/ml_dsa/dilithium_rounding.c
new file mode 100644
index 000000000000..c8a9c88185cf
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_rounding.c
@@ -0,0 +1,128 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#include "dilithium_rounding.h"
+
+/**
+ * @brief power2round - For finite field element a, compute a0, a1 such that
+ * a mod^+ Q = a1*2^D + a0 with -2^{D-1} < a0 <= 2^{D-1}.
+ * Assumes a to be standard representative.
+ *
+ * @param [in] a input element
+ * @param [out] a0 pointer to output element a0
+ *
+ * @return a1.
+ */
+int32_t power2round(int32_t *a0, int32_t a)
+{
+ int32_t a1;
+
+ a1 = (a + (1 << (DILITHIUM_D - 1)) - 1) >> DILITHIUM_D;
+ *a0 = a - (a1 << DILITHIUM_D);
+ return a1;
+}
+
+/**
+ * @brief decompose - For finite field element a, compute high and low bits a0,
+ * a1 such that a mod^+ Q = a1*ALPHA + a0 with
+ * -ALPHA/2 < a0 <= ALPHA/2 except if a1 = (Q-1)/ALPHA where
+ * we set a1 = 0 and -ALPHA/2 <= a0 = a mod^+ Q - Q < 0.
+ * Assumes a to be standard representative.
+ *
+ * @param [in] a input element
+ * @param [out] a0 pointer to output element a0
+ *
+ * @return a1.
+ */
+int32_t decompose(int32_t *a0, int32_t a)
+{
+ int32_t a1;
+
+ a1 = (a + 127) >> 7;
+#if DILITHIUM_GAMMA2 == (DILITHIUM_Q - 1) / 32
+ a1 = (a1 * 1025 + (1 << 21)) >> 22;
+ a1 &= 15;
+#elif DILITHIUM_GAMMA2 == (DILITHIUM_Q - 1) / 88
+ a1 = (a1 * 11275 + (1 << 23)) >> 24;
+ a1 ^= ((43 - a1) >> 31) & a1;
+#else
+#error "Uknown GAMMA2"
+#endif
+
+ *a0 = a - a1 * 2 * DILITHIUM_GAMMA2;
+ *a0 -= (((DILITHIUM_Q - 1) / 2 - *a0) >> 31) & DILITHIUM_Q;
+
+ return a1;
+}
+
+/**
+ * @brief make_hint - Compute hint bit indicating whether the low bits of the
+ * input element overflow into the high bits.
+ *
+ * @param a0 [in] low bits of input element
+ * @param a1 [in] high bits of input element
+ *
+ * @return 1 if overflow.
+ */
+int32_t make_hint(int32_t a0, int32_t a1)
+{
+ if (a0 > DILITHIUM_GAMMA2 || a0 < -DILITHIUM_GAMMA2 ||
+ (a0 == -DILITHIUM_GAMMA2 && a1 != 0))
+ return 1;
+
+ return 0;
+}
+
+/**
+ * @brief use_hint - Correct high bits according to hint.
+ *
+ * @param [in] a input element
+ * @param [in] hint hint bit
+ *
+ * @return corrected high bits.
+ */
+int32_t use_hint(int32_t a, int32_t hint)
+{
+ int32_t a0, a1;
+
+ a1 = decompose(&a0, a);
+ if (hint == 0)
+ return a1;
+
+#if DILITHIUM_GAMMA2 == (DILITHIUM_Q - 1) / 32
+ if (a0 > 0)
+ return (a1 + 1) & 15;
+ else
+ return (a1 - 1) & 15;
+#elif DILITHIUM_GAMMA2 == (DILITHIUM_Q - 1) / 88
+ if (a0 > 0)
+ return (a1 == 43) ? 0 : a1 + 1;
+ else
+ return (a1 == 0) ? 43 : a1 - 1;
+#else
+#error "Uknown GAMMA2"
+#endif
+}
diff --git a/crypto/ml_dsa/dilithium_rounding.h b/crypto/ml_dsa/dilithium_rounding.h
new file mode 100644
index 000000000000..928b041856e9
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_rounding.h
@@ -0,0 +1,45 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#ifndef DILITHIUM_ROUNDING_H
+#define DILITHIUM_ROUNDING_H
+
+#include "dilithium_type.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+int32_t power2round(int32_t *a0, int32_t a);
+int32_t decompose(int32_t *a0, int32_t a);
+int32_t make_hint(int32_t a0, int32_t a1);
+int32_t use_hint(int32_t a, int32_t hint);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* DILITHIUM_ROUNDING_H */
diff --git a/crypto/ml_dsa/dilithium_service_helpers.h b/crypto/ml_dsa/dilithium_service_helpers.h
new file mode 100644
index 000000000000..d671634e2040
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_service_helpers.h
@@ -0,0 +1,99 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+
+#ifndef DILITHIUM_SERVICE_HELPERS_H
+#define DILITHIUM_SERVICE_HELPERS_H
+
+/**
+ * @brief rej_uniform - Sample uniformly random coefficients in [0, Q-1] by
+ * performing rejection sampling on array of random bytes.
+ *
+ * @param [out] a pointer to output array (allocated)
+ * @param [in] len number of coefficients to be sampled
+ * @param [in] buf array of random bytes
+ * @param [in] buflen length of array of random bytes
+ *
+ * @return number of sampled coefficients. Can be smaller than len if not enough
+ * random bytes were given.
+ */
+static inline unsigned int rej_uniform(int32_t *a, unsigned int len,
+ const uint8_t *buf, unsigned int buflen)
+{
+ unsigned int ctr, pos;
+ uint32_t t;
+
+ ctr = pos = 0;
+ while (ctr < len && pos + 3 <= buflen) {
+ t = buf[pos++];
+ t |= (uint32_t)buf[pos++] << 8;
+ t |= (uint32_t)buf[pos++] << 16;
+ t &= 0x7FFFFF;
+
+ if (t < DILITHIUM_Q)
+ a[ctr++] = (int32_t)t;
+ }
+
+ return ctr;
+}
+
+/**
+ * @brief rej_eta - Sample uniformly random coefficients in [-ETA, ETA] by
+ * performing rejection sampling on array of random bytes.
+ *
+ * @param [out] a pointer to output array (allocated)
+ * @param [in] len number of coefficients to be sampled
+ * @param [in] buf array of random bytes
+ * @param [in] buflen length of array of random bytes
+ *
+ * @return number of sampled coefficients. Can be smaller than len if not enough
+ * random bytes were given.
+ */
+static inline unsigned int rej_eta(int32_t *a, unsigned int len,
+ const uint8_t *buf, unsigned int buflen)
+{
+ unsigned int ctr, pos;
+ int32_t t0, t1;
+
+ ctr = pos = 0;
+ while (ctr < len && pos < buflen) {
+ t0 = buf[pos] & 0x0F;
+ t1 = buf[pos++] >> 4;
+
+#if DILITHIUM_ETA == 2
+ if (t0 < 15) {
+ t0 = t0 - (205 * t0 >> 10) * 5;
+ a[ctr++] = 2 - t0;
+ }
+ if (t1 < 15 && ctr < len) {
+ t1 = t1 - (205 * t1 >> 10) * 5;
+ a[ctr++] = 2 - t1;
+ }
+#elif DILITHIUM_ETA == 4
+ if (t0 < 9)
+ a[ctr++] = 4 - t0;
+ if (t1 < 9 && ctr < len)
+ a[ctr++] = 4 - t1;
+#else
+#error "Undefined DILITHIUM_ETA"
+#endif
+ }
+
+ return ctr;
+}
+
+#endif /* DILITHIUM_SERVICE_HELPERS_H */
diff --git a/crypto/ml_dsa/dilithium_sig.c b/crypto/ml_dsa/dilithium_sig.c
new file mode 100644
index 000000000000..5495dd451de0
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_sig.c
@@ -0,0 +1,404 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+/*
+ * Copyright (C) 2024 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+
+#include <linux/init.h>
+#include <linux/module.h>
+#include <crypto/internal/sig.h>
+#include "dilithium.h"
+
+enum dilithium_kernel_key_type {
+ dilithium_kernel_key_unset = 0,
+ dilithium_kernel_key_sk = 1,
+ dilithium_kernel_key_pk = 2,
+};
+
+struct dilithium_kernel_ctx {
+ union {
+ struct dilithium_sk sk;
+ struct dilithium_pk pk;
+ };
+ enum dilithium_kernel_key_type key_type;
+};
+
+/* src -> message */
+/* dst -> signature */
+static int dilithium_kernel_sign(struct crypto_sig *tfm, const void *src,
+ unsigned int slen, void *dst,
+ unsigned int dlen)
+{
+ struct dilithium_kernel_ctx *ctx = crypto_sig_ctx(tfm);
+ struct dilithium_sig *sig;
+ enum dilithium_type type;
+ uint8_t *sig_ptr;
+ size_t sig_len;
+ int ret;
+
+ if (unlikely(ctx->key_type != dilithium_kernel_key_sk))
+ return -EINVAL;
+
+ type = dilithium_sk_type(&ctx->sk);
+ if (dlen != dilithium_sig_size(type))
+ return -EINVAL;
+
+ sig = kmalloc(sizeof(struct dilithium_sig), GFP_KERNEL);
+ if (!sig)
+ return -ENOMEM;
+
+ ret = dilithium_sign(sig, src, slen, &ctx->sk, lc_seeded_rng);
+ if (ret)
+ goto out;
+
+ ret = dilithium_sig_ptr(&sig_ptr, &sig_len, sig);
+ if (ret)
+ goto out;
+
+ memcpy(dst, sig_ptr, sig_len);
+ ret = sig_len;
+
+out:
+ kfree_sensitive(sig);
+ return ret;
+}
+
+/* src -> signature */
+/* msg -> message */
+static int dilithium_kernel_verify(struct crypto_sig *tfm, const void *src,
+ unsigned int slen, const void *msg,
+ unsigned int msg_len)
+{
+ struct dilithium_kernel_ctx *ctx = crypto_sig_ctx(tfm);
+ struct dilithium_sig *sig;
+ size_t sig_len;
+ enum dilithium_type type;
+ int ret;
+
+ if (unlikely(ctx->key_type != dilithium_kernel_key_pk))
+ return -EINVAL;
+
+ type = dilithium_pk_type(&ctx->pk);
+ sig_len = dilithium_sig_size(type);
+ if (slen < sig_len)
+ return -EINVAL;
+
+ sig = kmalloc(sizeof(struct dilithium_sig), GFP_KERNEL);
+ if (!sig)
+ return -ENOMEM;
+
+ ret = dilithium_sig_load(sig, src, sig_len);
+ if (ret)
+ goto out;
+
+ ret = dilithium_verify(sig, msg, msg_len, &ctx->pk);
+
+out:
+ kfree_sensitive(sig);
+ return ret;
+}
+
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+static unsigned int dilithium_kernel_87_key_size(struct crypto_sig *tfm)
+{
+ struct dilithium_kernel_ctx *ctx = crypto_sig_ctx(tfm);
+
+ switch (ctx->key_type) {
+ case dilithium_kernel_key_sk:
+ return sizeof(struct dilithium_87_sk);
+
+ case dilithium_kernel_key_unset:
+ case dilithium_kernel_key_pk:
+ default:
+ return sizeof(struct dilithium_87_pk);
+ }
+}
+#endif
+
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+static unsigned int dilithium_kernel_65_key_size(struct crypto_sig *tfm)
+{
+ struct dilithium_kernel_ctx *ctx = crypto_sig_ctx(tfm);
+
+ switch (ctx->key_type) {
+ case dilithium_kernel_key_sk:
+ return sizeof(struct dilithium_65_sk);
+
+ case dilithium_kernel_key_unset:
+ case dilithium_kernel_key_pk:
+ default:
+ return sizeof(struct dilithium_65_pk);
+ }
+}
+#endif
+
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+static unsigned int dilithium_kernel_44_key_size(struct crypto_sig *tfm)
+{
+ struct dilithium_kernel_ctx *ctx = crypto_sig_ctx(tfm);
+
+ switch (ctx->key_type) {
+ case dilithium_kernel_key_sk:
+ return sizeof(struct dilithium_44_sk);
+
+ case dilithium_kernel_key_unset:
+ case dilithium_kernel_key_pk:
+ default:
+ return sizeof(struct dilithium_44_pk);
+ }
+}
+#endif
+
+static int dilithium_kernel_set_pub_key_int(struct crypto_sig *tfm,
+ const void *key,
+ unsigned int keylen,
+ enum dilithium_type type)
+{
+ struct dilithium_kernel_ctx *ctx = crypto_sig_ctx(tfm);
+ int ret;
+
+ ctx->key_type = dilithium_kernel_key_unset;
+
+ ret = dilithium_pk_load(&ctx->pk, key, keylen);
+ if (!ret) {
+ if (dilithium_pk_type(&ctx->pk) != type)
+ ret = -EOPNOTSUPP;
+ else
+ ctx->key_type = dilithium_kernel_key_pk;
+ }
+
+ return ret;
+}
+
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+static int dilithium_kernel_44_set_pub_key(struct crypto_sig *tfm,
+ const void *key,
+ unsigned int keylen)
+{
+ return dilithium_kernel_set_pub_key_int(tfm, key, keylen, DILITHIUM_44);
+}
+#endif
+
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+static int dilithium_kernel_65_set_pub_key(struct crypto_sig *tfm,
+ const void *key,
+ unsigned int keylen)
+{
+ return dilithium_kernel_set_pub_key_int(tfm, key, keylen, DILITHIUM_65);
+}
+#endif
+
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+static int dilithium_kernel_87_set_pub_key(struct crypto_sig *tfm,
+ const void *key,
+ unsigned int keylen)
+{
+ return dilithium_kernel_set_pub_key_int(tfm, key, keylen, DILITHIUM_87);
+}
+#endif
+
+static int dilithium_kernel_set_priv_key_int(struct crypto_sig *tfm,
+ const void *key,
+ unsigned int keylen,
+ enum dilithium_type type)
+{
+ struct dilithium_kernel_ctx *ctx = crypto_sig_ctx(tfm);
+ int ret;
+
+ ctx->key_type = dilithium_kernel_key_unset;
+
+ ret = dilithium_sk_load(&ctx->sk, key, keylen);
+
+ if (!ret) {
+ if (dilithium_sk_type(&ctx->sk) != type)
+ ret = -EOPNOTSUPP;
+ else
+ ctx->key_type = dilithium_kernel_key_sk;
+ }
+
+ return ret;
+}
+
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+static int dilithium_kernel_44_set_priv_key(struct crypto_sig *tfm,
+ const void *key,
+ unsigned int keylen)
+{
+ return dilithium_kernel_set_priv_key_int(tfm, key, keylen,
+ DILITHIUM_44);
+}
+#endif
+
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+static int dilithium_kernel_65_set_priv_key(struct crypto_sig *tfm,
+ const void *key,
+ unsigned int keylen)
+{
+ return dilithium_kernel_set_priv_key_int(tfm, key, keylen,
+ DILITHIUM_65);
+}
+#endif
+
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+static int dilithium_kernel_87_set_priv_key(struct crypto_sig *tfm,
+ const void *key,
+ unsigned int keylen)
+{
+ return dilithium_kernel_set_priv_key_int(tfm, key, keylen,
+ DILITHIUM_87);
+}
+#endif
+
+static unsigned int dilithium_kernel_max_size(struct crypto_sig *tfm)
+{
+ struct dilithium_kernel_ctx *ctx = crypto_sig_ctx(tfm);
+ enum dilithium_type type;
+
+ switch (ctx->key_type) {
+ case dilithium_kernel_key_sk:
+ type = dilithium_sk_type(&ctx->sk);
+ /* When SK is set -> generate a signature */
+ return dilithium_sig_size(type);
+ case dilithium_kernel_key_pk:
+ type = dilithium_pk_type(&ctx->pk);
+ /* When PK is set, this is a safety valve, result is boolean */
+ return dilithium_sig_size(type);
+ default:
+ return 0;
+ }
+}
+
+static int dilithium_kernel_alg_init(struct crypto_sig *tfm)
+{
+ return 0;
+}
+
+static void dilithium_kernel_alg_exit(struct crypto_sig *tfm)
+{
+ struct dilithium_kernel_ctx *ctx = crypto_sig_ctx(tfm);
+
+ ctx->key_type = dilithium_kernel_key_unset;
+}
+
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+static struct sig_alg dilithium_kernel_87 = {
+ .sign = dilithium_kernel_sign,
+ .verify = dilithium_kernel_verify,
+ .set_pub_key = dilithium_kernel_87_set_pub_key,
+ .set_priv_key = dilithium_kernel_87_set_priv_key,
+ .key_size = dilithium_kernel_87_key_size,
+ .max_size = dilithium_kernel_max_size,
+ .init = dilithium_kernel_alg_init,
+ .exit = dilithium_kernel_alg_exit,
+ .base.cra_name = "ml-dsa87",
+ .base.cra_driver_name = "ml-dsa87-leancrypto",
+ .base.cra_ctxsize = sizeof(struct dilithium_kernel_ctx),
+ .base.cra_module = THIS_MODULE,
+ .base.cra_priority = 5000,
+};
+#endif
+
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+static struct sig_alg dilithium_kernel_65 = {
+ .sign = dilithium_kernel_sign,
+ .verify = dilithium_kernel_verify,
+ .set_pub_key = dilithium_kernel_65_set_pub_key,
+ .set_priv_key = dilithium_kernel_65_set_priv_key,
+ .key_size = dilithium_kernel_65_key_size,
+ .max_size = dilithium_kernel_max_size,
+ .init = dilithium_kernel_alg_init,
+ .exit = dilithium_kernel_alg_exit,
+ .base.cra_name = "ml-dsa65",
+ .base.cra_driver_name = "ml-dsa65-leancrypto",
+ .base.cra_ctxsize = sizeof(struct dilithium_kernel_ctx),
+ .base.cra_module = THIS_MODULE,
+ .base.cra_priority = 5000,
+};
+#endif
+
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+static struct sig_alg dilithium_kernel_44 = {
+ .sign = dilithium_kernel_sign,
+ .verify = dilithium_kernel_verify,
+ .set_pub_key = dilithium_kernel_44_set_pub_key,
+ .set_priv_key = dilithium_kernel_44_set_priv_key,
+ .key_size = dilithium_kernel_44_key_size,
+ .max_size = dilithium_kernel_max_size,
+ .init = dilithium_kernel_alg_init,
+ .exit = dilithium_kernel_alg_exit,
+ .base.cra_name = "ml-dsa44",
+ .base.cra_driver_name = "ml-dsa44-leancrypto",
+ .base.cra_ctxsize = sizeof(struct dilithium_kernel_ctx),
+ .base.cra_module = THIS_MODULE,
+ .base.cra_priority = 5000,
+};
+#endif
+
+static int __init dilithium_init(void)
+{
+ int ret;
+
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ ret = crypto_register_sig(&dilithium_kernel_44);
+ if (ret < 0)
+ goto error_44;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ ret = crypto_register_sig(&dilithium_kernel_65);
+ if (ret < 0)
+ goto error_65;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ ret = crypto_register_sig(&dilithium_kernel_87);
+ if (ret < 0)
+ goto error_87;
+#endif
+ return 0;
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+error_87:
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ crypto_unregister_sig(&dilithium_kernel_65);
+error_65:
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ crypto_unregister_sig(&dilithium_kernel_44);
+error_44:
+#endif
+ pr_err("Failed to register (%d)\n", ret);
+ return ret;
+}
+module_init(dilithium_init);
+
+static void dilithium_exit(void)
+{
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ crypto_unregister_sig(&dilithium_kernel_87);
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ crypto_unregister_sig(&dilithium_kernel_65);
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ crypto_unregister_sig(&dilithium_kernel_44);
+#endif
+}
+module_exit(dilithium_exit);
+
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_DESCRIPTION("Leancrypto ML-DSA/Dilithium");
+MODULE_AUTHOR("Stephan Mueller <smueller@chronox.de>");
+MODULE_ALIAS_CRYPTO("ml-dsa44");
+MODULE_ALIAS_CRYPTO("ml-dsa65");
+MODULE_ALIAS_CRYPTO("ml-dsa87");
diff --git a/crypto/ml_dsa/dilithium_signature_c.c b/crypto/ml_dsa/dilithium_signature_c.c
new file mode 100644
index 000000000000..a4a9d0bfdd37
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_signature_c.c
@@ -0,0 +1,174 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#include "dilithium_signature_c.h"
+
+/* We need once the buffer size to handle the hashing */
+#define LC_POLY_UNIFOR_BUF_SIZE_MULTIPLIER 1
+
+#include "dilithium_poly.h"
+#include "dilithium_poly_common.h"
+#include "dilithium_poly_c.h"
+#include "dilithium_polyvec.h"
+#include "dilithium_polyvec_c.h"
+#include "dilithium_pack.h"
+#include "dilithium_signature_impl.h"
+
+int dilithium_sign_c(struct dilithium_sig *sig,
+ const uint8_t *m, size_t mlen,
+ const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx)
+{
+ return dilithium_sign_impl(sig, m, mlen, sk, rng_ctx);
+}
+
+int dilithium_sign_ctx_c(struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen, const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx)
+{
+ return dilithium_sign_ctx_impl(sig, ctx, m, mlen, sk, rng_ctx);
+}
+
+int dilithium_sign_init_c(struct dilithium_ctx *ctx,
+ const struct dilithium_sk *sk)
+{
+ return dilithium_sign_init_impl(ctx, sk);
+}
+
+int dilithium_sign_update_c(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen)
+{
+ return dilithium_sign_update_impl(ctx, m, mlen);
+}
+
+int dilithium_sign_final_c(struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx)
+{
+ return dilithium_sign_final_impl(sig, ctx, sk, rng_ctx);
+}
+
+int dilithium_verify_c(const struct dilithium_sig *sig, const uint8_t *m,
+ size_t mlen, const struct dilithium_pk *pk)
+{
+ return dilithium_verify_impl(sig, m, mlen, pk);
+}
+
+int dilithium_verify_ctx_c(const struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen, const struct dilithium_pk *pk)
+{
+ return dilithium_verify_ctx_impl(sig, ctx, m, mlen, pk);
+}
+
+int dilithium_verify_init_c(struct dilithium_ctx *ctx,
+ const struct dilithium_pk *pk)
+{
+ return dilithium_verify_init_impl(ctx, pk);
+}
+
+int dilithium_verify_update_c(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen)
+{
+ return dilithium_verify_update_impl(ctx, m, mlen);
+}
+
+int dilithium_verify_final_c(const struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_pk *pk)
+{
+ return dilithium_verify_final_impl(sig, ctx, pk);
+}
+
+int dilithium_sign(struct dilithium_sig *sig,
+ const uint8_t *m, size_t mlen,
+ const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx)
+{
+ return dilithium_sign_c(sig, m, mlen, sk, rng_ctx);
+}
+
+int dilithium_sign_ctx(struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen, const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx)
+{
+ return dilithium_sign_ctx_c(sig, ctx, m, mlen, sk, rng_ctx);
+}
+
+int dilithium_sign_init(struct dilithium_ctx *ctx,
+ const struct dilithium_sk *sk)
+{
+ return dilithium_sign_init_c(ctx, sk);
+}
+
+int dilithium_sign_update(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen)
+{
+ return dilithium_sign_update_c(ctx, m, mlen);
+}
+
+int dilithium_sign_final(struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx)
+{
+ return dilithium_sign_final_c(sig, ctx, sk, rng_ctx);
+}
+
+int dilithium_verify(const struct dilithium_sig *sig, const uint8_t *m,
+ size_t mlen, const struct dilithium_pk *pk)
+{
+ return dilithium_verify_c(sig, m, mlen, pk);
+}
+
+int dilithium_verify_ctx(const struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen, const struct dilithium_pk *pk)
+{
+ return dilithium_verify_ctx_c(sig, ctx, m, mlen, pk);
+}
+
+int dilithium_verify_init(struct dilithium_ctx *ctx,
+ const struct dilithium_pk *pk)
+{
+ return dilithium_verify_init_c(ctx, pk);
+}
+
+int dilithium_verify_update(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen)
+{
+ return dilithium_verify_update_c(ctx, m, mlen);
+}
+
+int dilithium_verify_final(const struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_pk *pk)
+{
+ return dilithium_verify_final_c(sig, ctx, pk);
+}
diff --git a/crypto/ml_dsa/dilithium_signature_c.h b/crypto/ml_dsa/dilithium_signature_c.h
new file mode 100644
index 000000000000..a7d20cd49672
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_signature_c.h
@@ -0,0 +1,53 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+
+#ifndef DILITHIUM_SIGNATURE_C_H
+#define DILITHIUM_SIGNATURE_C_H
+
+#include "dilithium_type.h"
+
+int dilithium_sign_c(struct dilithium_sig *sig, const uint8_t *m,
+ size_t mlen, const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx);
+int dilithium_sign_ctx_c(struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen, const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx);
+int dilithium_sign_init_c(struct dilithium_ctx *ctx,
+ const struct dilithium_sk *sk);
+int dilithium_sign_update_c(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen);
+int dilithium_sign_final_c(struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx);
+
+int dilithium_verify_c(const struct dilithium_sig *sig, const uint8_t *m,
+ size_t mlen, const struct dilithium_pk *pk);
+int dilithium_verify_ctx_c(const struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen, const struct dilithium_pk *pk);
+int dilithium_verify_init_c(struct dilithium_ctx *ctx,
+ const struct dilithium_pk *pk);
+int dilithium_verify_update_c(struct dilithium_ctx *ctx, const uint8_t *m,
+ size_t mlen);
+int dilithium_verify_final_c(const struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_pk *pk);
+
+#endif /* DILITHIUM_SIGNATURE_C_H */
diff --git a/crypto/ml_dsa/dilithium_signature_helper.c b/crypto/ml_dsa/dilithium_signature_helper.c
new file mode 100644
index 000000000000..3f6b9d02602f
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_signature_helper.c
@@ -0,0 +1,110 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * Copyright (C) 2024 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+
+#include <linux/export.h>
+#include <linux/math.h>
+#include <linux/slab.h>
+#include "dilithium_type.h"
+
+#define cround(x) round_up((x), umax(DILITHIUM_AHAT_ALIGNMENT, CRYPTO_MINALIGN))
+
+struct dilithium_ctx *dilithium_ctx_alloc(void)
+{
+ struct dilithium_ctx *ctx;
+ struct crypto_shash *shash;
+ void *p;
+
+ shash = crypto_alloc_shash("shake256", 0, 0);
+ if (IS_ERR(shash)) {
+ pr_warn("no shake256: %ld\n", PTR_ERR(shash));
+ return ERR_CAST(shash);
+ }
+
+ p = kzalloc(cround(sizeof(struct dilithium_ctx)) +
+ cround(crypto_shash_descsize(shash)),
+ GFP_KERNEL);
+ if (!p) {
+ crypto_free_shash(shash);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ ctx = p;
+ //ctx->dilithium_hash_ctx = p + cround(sizeof(struct dilithium_ctx));
+ ctx->dilithium_prehash_type = shash;
+ return ctx;
+}
+EXPORT_SYMBOL(dilithium_ctx_alloc);
+
+struct dilithium_ctx *dilithium_ctx_alloc_ahat(enum dilithium_type type)
+{
+ struct dilithium_ctx *ctx;
+ struct crypto_shash *shash;
+ size_t ahat_size;
+ void *p;
+
+ shash = crypto_alloc_shash("shake256", 0, 0);
+ if (IS_ERR(shash))
+ return ERR_CAST(shash);
+
+ switch (type) {
+#ifdef CONFIG_CRYPTO_DILITHIUM_44
+ case DILITHIUM_44:
+ ahat_size = DILITHIUM_44_AHAT_SIZE;
+ break;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_65
+ case DILITHIUM_65:
+ ahat_size = DILITHIUM_65_AHAT_SIZE;
+ break;
+#endif
+#ifdef CONFIG_CRYPTO_DILITHIUM_87
+ case DILITHIUM_87:
+ ahat_size = DILITHIUM_87_AHAT_SIZE;
+ break;
+#endif
+ default:
+ WARN_ON(1);
+ return ERR_PTR(-EINVAL);
+ }
+
+ p = kzalloc(cround(sizeof(struct dilithium_ctx)) +
+ cround(ahat_size) +
+ cround(crypto_shash_descsize(shash)),
+ GFP_KERNEL);
+ if (!p) {
+ crypto_free_shash(shash);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ ctx = p;
+ p += cround(sizeof(struct dilithium_ctx));
+ ctx->ahat = p;
+ ctx->ahat_size = ahat_size;
+ p += cround(ahat_size);
+ //ctx->dilithium_hash_ctx = p;
+ ctx->dilithium_prehash_type = shash;
+ return ctx;
+}
+EXPORT_SYMBOL(dilithium_ctx_alloc_ahat);
+
+void dilithium_ctx_zero_free(struct dilithium_ctx *ctx)
+{
+ crypto_free_shash(ctx->dilithium_prehash_type);
+ kfree_sensitive(ctx);
+}
+EXPORT_SYMBOL(dilithium_ctx_zero_free);
diff --git a/crypto/ml_dsa/dilithium_signature_impl.h b/crypto/ml_dsa/dilithium_signature_impl.h
new file mode 100644
index 000000000000..1f0ec0f7c27c
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_signature_impl.h
@@ -0,0 +1,838 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#ifndef DILITHIUM_SIGNATURE_IMPL_H
+#define DILITHIUM_SIGNATURE_IMPL_H
+
+#include "dilithium.h"
+#include "dilithium_type.h"
+#include "dilithium_debug.h"
+#include "dilithium_pack.h"
+#include "dilithium_signature_impl.h"
+#include "signature_domain_separation.h"
+#include "small_stack_support.h"
+
+/*
+ * Enable this macro to report the rejection code paths taken with the
+ * signature generation operation. When disabled, the compiler should
+ * eliminate this code which means that the counting code is folded away.
+ */
+#undef REJECTION_TEST_SAMPLING
+
+#define _WS_POLY_UNIFORM_BUF_SIZE \
+ (POLY_UNIFORM_NBLOCKS * SHAKE128_BLOCK_SIZE + 2)
+
+#ifndef LC_POLY_UNIFOR_BUF_SIZE_MULTIPLIER
+#error "LC_POLY_UNIFOR_BUF_SIZE_MULTIPLIER is not defined"
+#endif
+
+#define WS_POLY_UNIFORM_BUF_SIZE \
+ (_WS_POLY_UNIFORM_BUF_SIZE * LC_POLY_UNIFOR_BUF_SIZE_MULTIPLIER)
+
+static int dilithium_sign_internal_ahat(struct dilithium_sig *sig,
+ const struct dilithium_sk *sk,
+ struct dilithium_ctx *ctx,
+ struct crypto_rng *rng_ctx)
+{
+ struct workspace_sign {
+ polyvecl s1, y, z;
+ polyveck t0, s2, w1, w0, h;
+ poly cp;
+ uint8_t seedbuf[DILITHIUM_SEEDBYTES + DILITHIUM_RNDBYTES +
+ DILITHIUM_CRHBYTES];
+ union {
+ uint8_t poly_uniform_gamma1_buf[WS_POLY_UNIFORM_BUF_SIZE];
+ uint8_t poly_challenge_buf[POLY_CHALLENGE_BYTES];
+ } tmp;
+ };
+ unsigned int n;
+ uint8_t *key, *mu, *rhoprime, *rnd;
+ const polyvecl *mat = ctx->ahat;
+ uint16_t nonce = 0;
+ int ret = 0;
+ uint8_t __maybe_unused rej_total = 0;
+ LC_DECLARE_MEM(ws, struct workspace_sign, sizeof(uint64_t));
+
+ /* AHat must be present at this time */
+ if (WARN_ON_ONCE(!mat)) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ key = ws->seedbuf;
+ rnd = key + DILITHIUM_SEEDBYTES;
+ mu = rnd + DILITHIUM_RNDBYTES;
+
+ /*
+ * If the external mu is provided, use this verbatim, otherwise
+ * calculate the mu value.
+ */
+ if (ctx->external_mu) {
+ if (ctx->external_mu_len != DILITHIUM_CRHBYTES)
+ return -EINVAL;
+ memcpy(mu, ctx->external_mu, DILITHIUM_CRHBYTES);
+ } else {
+ /*
+ * Set the digestsize - for SHA512 this is a noop, for SHAKE256,
+ * it sets the value. The BUILD_BUG_ON is to check that the
+ * SHA-512 output size is identical to the expected length.
+ */
+ BUILD_BUG_ON(DILITHIUM_CRHBYTES != SHA3_512_DIGEST_SIZE);
+ if (!dilithium_hash_check_digestsize(ctx, DILITHIUM_CRHBYTES)) {
+ ret = -EINVAL;
+ goto out;
+ }
+ dilithium_hash_final(ctx, mu, DILITHIUM_CRHBYTES);
+ }
+ dilithium_print_buffer(mu, DILITHIUM_CRHBYTES, "Siggen - MU:");
+
+ if (rng_ctx) {
+ ret = crypto_rng_generate(rng_ctx, NULL, 0, rnd,
+ DILITHIUM_RNDBYTES);
+ if (ret < 0)
+ goto out;
+ } else {
+ memset(rnd, 0, DILITHIUM_RNDBYTES);
+ }
+ dilithium_print_buffer(rnd, DILITHIUM_RNDBYTES, "Siggen - RND:");
+
+ unpack_sk_key(key, sk);
+
+ /* Timecop: key is secret */
+ poison(key, DILITHIUM_SEEDBYTES);
+
+ /* Re-use the ws->seedbuf, but making sure that mu is unchanged */
+ BUILD_BUG_ON(DILITHIUM_CRHBYTES >
+ DILITHIUM_SEEDBYTES + DILITHIUM_RNDBYTES);
+ rhoprime = key;
+
+ shake256(key,
+ DILITHIUM_SEEDBYTES + DILITHIUM_RNDBYTES +
+ DILITHIUM_CRHBYTES,
+ rhoprime, DILITHIUM_CRHBYTES);
+ dilithium_print_buffer(rhoprime, DILITHIUM_CRHBYTES,
+ "Siggen - RHOPrime:");
+
+ /*
+ * Timecop: RHO' is the hash of the secret value of key which is
+ * enlarged to sample the intermediate vector y from. Due to the hashing
+ * any side channel on RHO' cannot allow the deduction of the original
+ * key.
+ */
+ unpoison(rhoprime, DILITHIUM_CRHBYTES);
+
+ unpack_sk_s1(&ws->s1, sk);
+
+ /* Timecop: s1 is secret */
+ poison(&ws->s1, sizeof(polyvecl));
+
+ polyvecl_ntt(&ws->s1);
+ dilithium_print_polyvecl(&ws->s1,
+ "Siggen - S1 L x N matrix after NTT:");
+
+ unpack_sk_s2(&ws->s2, sk);
+
+ /* Timecop: s2 is secret */
+ poison(&ws->s2, sizeof(polyveck));
+
+ polyveck_ntt(&ws->s2);
+ dilithium_print_polyveck(&ws->s2,
+ "Siggen - S2 K x N matrix after NTT:");
+
+ unpack_sk_t0(&ws->t0, sk);
+ polyveck_ntt(&ws->t0);
+ dilithium_print_polyveck(&ws->t0,
+ "Siggen - T0 K x N matrix after NTT:");
+
+rej:
+ /* Sample intermediate vector y */
+ polyvecl_uniform_gamma1(&ws->y, rhoprime, nonce++,
+ ws->tmp.poly_uniform_gamma1_buf);
+ dilithium_print_polyvecl(
+ &ws->y,
+ "Siggen - Y L x N matrix after ExpandMask - start of loop");
+
+ /* Timecop: s2 is secret */
+ poison(&ws->y, sizeof(polyvecl));
+
+ /* Matrix-vector multiplication */
+ ws->z = ws->y;
+ polyvecl_ntt(&ws->z);
+
+ /* Use the cp for this operation as it is not used here so far. */
+ polyvec_matrix_pointwise_montgomery(&ws->w1, mat, &ws->z, &ws->cp);
+ polyveck_reduce(&ws->w1);
+ polyveck_invntt_tomont(&ws->w1);
+ dilithium_print_polyveck(&ws->w1,
+ "Siggen - W K x N matrix after NTT-1");
+
+ /* Decompose w and call the random oracle */
+ polyveck_caddq(&ws->w1);
+ polyveck_decompose(&ws->w1, &ws->w0, &ws->w1);
+
+ /* Timecop: the signature component w1 is not sensitive any more. */
+ unpoison(&ws->w1, sizeof(polyveck));
+ polyveck_pack_w1(sig->sig, &ws->w1);
+ dilithium_print_buffer(sig->sig,
+ DILITHIUM_K * DILITHIUM_POLYW1_PACKEDBYTES,
+ "Siggen - w1Encode of W1");
+
+ dilithium_hash_init(ctx);
+ dilithium_hash_update(ctx, mu, DILITHIUM_CRHBYTES);
+ dilithium_hash_finup(ctx, sig->sig,
+ DILITHIUM_K * DILITHIUM_POLYW1_PACKEDBYTES,
+ sig->sig, DILITHIUM_CTILDE_BYTES);
+
+ dilithium_print_buffer(sig->sig, DILITHIUM_CTILDE_BYTES,
+ "Siggen - ctilde");
+
+ poly_challenge(&ws->cp, sig->sig, ws->tmp.poly_challenge_buf);
+ dilithium_print_poly(&ws->cp, "Siggen - c after SampleInBall");
+ poly_ntt(&ws->cp);
+ dilithium_print_poly(&ws->cp, "Siggen - c after NTT");
+
+ /* Compute z, reject if it reveals secret */
+ polyvecl_pointwise_poly_montgomery(&ws->z, &ws->cp, &ws->s1);
+ polyvecl_invntt_tomont(&ws->z);
+ polyvecl_add(&ws->z, &ws->z, &ws->y);
+ dilithium_print_polyvecl(&ws->z, "Siggen - z <- y + cs1");
+
+ polyvecl_reduce(&ws->z);
+ dilithium_print_polyvecl(&ws->z, "Siggen - z reduction");
+
+ /* Timecop: the signature component z is not sensitive any more. */
+ unpoison(&ws->z, sizeof(polyvecl));
+
+ if (polyvecl_chknorm(&ws->z, DILITHIUM_GAMMA1 - DILITHIUM_BETA)) {
+ dilithium_print_polyvecl(&ws->z, "Siggen - z rejection");
+ rej_total |= 1 << 0;
+ goto rej;
+ }
+
+ /*
+ * Check that subtracting cs2 does not change high bits of w and low
+ * bits do not reveal secret information.
+ */
+ polyveck_pointwise_poly_montgomery(&ws->h, &ws->cp, &ws->s2);
+ polyveck_invntt_tomont(&ws->h);
+ polyveck_sub(&ws->w0, &ws->w0, &ws->h);
+ polyveck_reduce(&ws->w0);
+
+ /* Timecop: verification data w0 is not sensitive any more. */
+ unpoison(&ws->w0, sizeof(polyveck));
+
+ if (polyveck_chknorm(&ws->w0,
+ DILITHIUM_GAMMA2 - DILITHIUM_BETA)) {
+ dilithium_print_polyveck(&ws->w0, "Siggen - r0 rejection");
+ rej_total |= 1 << 1;
+ goto rej;
+ }
+
+ /* Compute hints for w1 */
+ polyveck_pointwise_poly_montgomery(&ws->h, &ws->cp, &ws->t0);
+ polyveck_invntt_tomont(&ws->h);
+ polyveck_reduce(&ws->h);
+
+ /* Timecop: the signature component h is not sensitive any more. */
+ unpoison(&ws->h, sizeof(polyveck));
+
+ if (polyveck_chknorm(&ws->h, DILITHIUM_GAMMA2)) {
+ dilithium_print_polyveck(&ws->h, "Siggen - ct0 rejection");
+ rej_total |= 1 << 2;
+ goto rej;
+ }
+
+ polyveck_add(&ws->w0, &ws->w0, &ws->h);
+
+ n = polyveck_make_hint(&ws->h, &ws->w0, &ws->w1);
+ if (n > DILITHIUM_OMEGA) {
+ dilithium_print_polyveck(&ws->w0, "Siggen - h rejection");
+ rej_total |= 1 << 3;
+ goto rej;
+ }
+
+ /* Write signature */
+ dilithium_print_buffer(sig->sig, DILITHIUM_CTILDE_BYTES,
+ "Siggen - Ctilde:");
+ dilithium_print_polyvecl(&ws->z, "Siggen - Z L x N matrix:");
+ dilithium_print_polyveck(&ws->h, "Siggen - H K x N matrix:");
+
+ pack_sig(sig, &ws->z, &ws->h);
+
+ dilithium_print_buffer(sig->sig, DILITHIUM_CRYPTO_BYTES,
+ "Siggen - Signature:");
+
+out:
+ LC_RELEASE_MEM(ws);
+#ifdef REJECTION_TEST_SAMPLING
+ return ret ? ret : rej_total;
+#else
+ return ret;
+#endif
+}
+
+static int dilithium_sign_internal_noahat(struct dilithium_sig *sig,
+ const struct dilithium_sk *sk,
+ struct dilithium_ctx *ctx,
+ struct crypto_rng *rng_ctx)
+{
+ struct workspace_sign {
+ polyvecl mat[DILITHIUM_K];
+ uint8_t poly_uniform_buf[WS_POLY_UNIFORM_BUF_SIZE];
+ };
+ /* The first bytes of the key is rho. */
+ const uint8_t *rho = sk->sk;
+ int ret = 0;
+ LC_DECLARE_MEM(ws, struct workspace_sign, DILITHIUM_AHAT_ALIGNMENT);
+
+ polyvec_matrix_expand(ws->mat, rho, ws->poly_uniform_buf);
+
+ /* Temporarily set the pointer */
+ ctx->ahat = ws->mat;
+
+ ret = dilithium_sign_internal_ahat(sig, sk, ctx, rng_ctx);
+
+ ctx->ahat = NULL;
+ LC_RELEASE_MEM(ws);
+ return ret;
+}
+
+static int dilithium_sk_expand_impl(const struct dilithium_sk *sk,
+ struct dilithium_ctx *ctx)
+{
+ struct workspace_sign {
+ uint8_t poly_uniform_buf[WS_POLY_UNIFORM_BUF_SIZE];
+ };
+ /* The first bytes of the key is rho. */
+ const uint8_t *rho = sk->sk;
+ polyvecl *mat = ctx->ahat;
+ int ret = 0;
+ LC_DECLARE_MEM(ws, struct workspace_sign, sizeof(uint64_t));
+
+ /*
+ * The compile time sanity check links API header file with
+ * Dilithium-internal definitions.
+ *
+ * Runtime sanity check ensures that the allocated context has
+ * sufficient size (e.g. not that caller used, say,
+ * DILITHIUM_44_CTX_ON_STACK_AHAT with a ML-DSA 65 or 87 key)
+ */
+#if DILITHIUM_MODE == 2
+ BUILD_BUG_ON(DILITHIUM_44_AHAT_SIZE !=
+ sizeof(polyvecl) * DILITHIUM44_K);
+ if (ctx->ahat_size < DILITHIUM_44_AHAT_SIZE) {
+ ret = -EOVERFLOW;
+ goto out;
+ }
+#elif DILITHIUM_MODE == 3
+ BUILD_BUG_ON(DILITHIUM_65_AHAT_SIZE !=
+ sizeof(polyvecl) * DILITHIUM65_K);
+ if (ctx->ahat_size < DILITHIUM_65_AHAT_SIZE) {
+ ret = -EOVERFLOW;
+ goto out;
+ }
+#elif DILITHIUM_MODE == 5
+ BUILD_BUG_ON(DILITHIUM_87_AHAT_SIZE !=
+ sizeof(polyvecl) * DILITHIUM87_K);
+ if (ctx->ahat_size < DILITHIUM_87_AHAT_SIZE) {
+ ret = -EOVERFLOW;
+ goto out;
+ }
+#else
+#error "Undefined DILITHIUM_MODE"
+#endif
+
+ polyvec_matrix_expand(mat, rho, ws->poly_uniform_buf);
+ dilithium_print_polyvecl_k(mat,
+ "AHAT - A K x L x N matrix after ExpandA:");
+
+ ctx->ahat_expanded = 1;
+
+out:
+ LC_RELEASE_MEM(ws);
+ return ret;
+}
+
+static int dilithium_sign_internal(struct dilithium_sig *sig,
+ const struct dilithium_sk *sk,
+ struct dilithium_ctx *ctx,
+ struct crypto_rng *rng_ctx)
+{
+ int ret;
+
+ if (!ctx->ahat)
+ return dilithium_sign_internal_noahat(sig, sk, ctx, rng_ctx);
+
+ if (!ctx->ahat_expanded) {
+ ret = dilithium_sk_expand_impl(sk, ctx);
+ if (ret < 0)
+ goto out;
+ }
+
+ ret = dilithium_sign_internal_ahat(sig, sk, ctx, rng_ctx);
+
+out:
+ return ret;
+}
+
+static int dilithium_sign_ctx_impl(struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx,
+ const uint8_t *m, size_t mlen,
+ const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx)
+{
+ uint8_t tr[DILITHIUM_TRBYTES];
+ int ret = 0;
+
+ /* rng_ctx is allowed to be NULL as handled below */
+ if (!sig || !sk || !ctx)
+ return -EINVAL;
+ /* Either the message or the external mu must be provided */
+ if (!m && !ctx->external_mu)
+ return -EINVAL;
+
+ dilithium_print_buffer(m, mlen, "Siggen - Message");
+
+ unpack_sk_tr(tr, sk);
+
+ if (m) {
+ /* Compute mu = CRH(tr, msg) */
+ dilithium_hash_init(ctx);
+ dilithium_hash_update(ctx, tr, DILITHIUM_TRBYTES);
+
+ ret = signature_domain_separation(
+ &ctx->dilithium_hash_ctx, ctx->ml_dsa_internal,
+ ctx->userctx, ctx->userctxlen,
+ m, mlen,
+ ctx->randomizer, ctx->randomizerlen,
+ DILITHIUM_NIST_CATEGORY);
+ if (ret < 0)
+ goto out;
+ }
+
+ ret = dilithium_sign_internal(sig, sk, ctx, rng_ctx);
+
+out:
+ memzero_explicit(tr, sizeof(tr));
+ return ret;
+}
+
+static int dilithium_sign_impl(struct dilithium_sig *sig,
+ const uint8_t *m, size_t mlen,
+ const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx)
+{
+ struct dilithium_ctx *ctx;
+ int ret;
+
+ ctx = dilithium_ctx_alloc();
+ if (IS_ERR(ctx))
+ return PTR_ERR(ctx);
+
+ ret = dilithium_sign_ctx_impl(sig, ctx, m, mlen, sk, rng_ctx);
+
+ dilithium_ctx_zero_free(ctx);
+ return ret;
+}
+
+static int dilithium_sign_init_impl(struct dilithium_ctx *ctx,
+ const struct dilithium_sk *sk)
+{
+ uint8_t tr[DILITHIUM_TRBYTES];
+
+ /* rng_ctx is allowed to be NULL as handled below */
+ if (!ctx || !sk)
+ return -EINVAL;
+
+ /* Require the use of SHAKE256 */
+ if (!dilithium_hash_check_blocksize(ctx, SHAKE256_BLOCK_SIZE))
+ return -EOPNOTSUPP;
+
+ unpack_sk_tr(tr, sk);
+
+ /* Compute mu = CRH(tr, msg) */
+ dilithium_hash_init(ctx);
+ dilithium_hash_update(ctx, tr, DILITHIUM_TRBYTES);
+ memzero_explicit(tr, sizeof(tr));
+
+ return signature_domain_separation(
+ &ctx->dilithium_hash_ctx, ctx->ml_dsa_internal,
+ ctx->userctx, ctx->userctxlen,
+ NULL, 0,
+ ctx->randomizer, ctx->randomizerlen,
+ DILITHIUM_NIST_CATEGORY);
+}
+
+static int dilithium_sign_update_impl(struct dilithium_ctx *ctx,
+ const uint8_t *m, size_t mlen)
+{
+ if (!ctx || !m)
+ return -EINVAL;
+
+ /* Compute CRH(tr, msg) */
+ dilithium_hash_update(ctx, m, mlen);
+
+ return 0;
+}
+
+static int dilithium_sign_final_impl(struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_sk *sk,
+ struct crypto_rng *rng_ctx)
+{
+ int ret = 0;
+
+ /* rng_ctx is allowed to be NULL as handled below */
+ if (!sig || !ctx || !sk) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ ret = dilithium_sign_internal(sig, sk, ctx, rng_ctx);
+
+out:
+ dilithium_ctx_zero(ctx);
+ return ret;
+}
+
+static int dilithium_verify_internal_ahat(const struct dilithium_sig *sig,
+ const struct dilithium_pk *pk,
+ struct dilithium_ctx *ctx)
+{
+ struct workspace_verify {
+ union {
+ poly cp;
+ } matrix;
+ polyveck w1;
+ union {
+ polyveck t1, h;
+ polyvecl z;
+ uint8_t mu[DILITHIUM_CRHBYTES];
+ union {
+ uint8_t coeffs[round_up(DILITHIUM_CTILDE_BYTES, 8)];
+ } __aligned(8) c2;
+ } buf;
+
+ union {
+ poly polyvecl_pointwise_acc_montgomery_buf;
+ uint8_t buf[DILITHIUM_K *
+ DILITHIUM_POLYW1_PACKEDBYTES];
+ uint8_t poly_challenge_buf[POLY_CHALLENGE_BYTES];
+ } tmp;
+ };
+ /* The first bytes of the signature is c~ and thus contains c1. */
+ const uint8_t *c1 = sig->sig;
+ const polyvecl *mat = ctx->ahat;
+ int ret = 0;
+ LC_DECLARE_MEM(ws, struct workspace_verify, sizeof(uint64_t));
+
+ /* AHat must be present at this time */
+ if (!mat) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ unpack_sig_z(&ws->buf.z, sig);
+ if (polyvecl_chknorm(&ws->buf.z,
+ DILITHIUM_GAMMA1 - DILITHIUM_BETA)) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ polyvecl_ntt(&ws->buf.z);
+ polyvec_matrix_pointwise_montgomery(
+ &ws->w1, mat, &ws->buf.z,
+ &ws->tmp.polyvecl_pointwise_acc_montgomery_buf);
+
+ /* Matrix-vector multiplication; compute Az - c2^dt1 */
+ poly_challenge(&ws->matrix.cp, c1, ws->tmp.poly_challenge_buf);
+ poly_ntt(&ws->matrix.cp);
+
+ unpack_pk_t1(&ws->buf.t1, pk);
+ polyveck_shiftl(&ws->buf.t1);
+ polyveck_ntt(&ws->buf.t1);
+ polyveck_pointwise_poly_montgomery(&ws->buf.t1, &ws->matrix.cp,
+ &ws->buf.t1);
+
+ polyveck_sub(&ws->w1, &ws->w1, &ws->buf.t1);
+ polyveck_reduce(&ws->w1);
+ polyveck_invntt_tomont(&ws->w1);
+
+ /* Reconstruct w1 */
+ polyveck_caddq(&ws->w1);
+ dilithium_print_polyveck(&ws->w1,
+ "Sigver - W K x N matrix before hint:");
+
+ if (unpack_sig_h(&ws->buf.h, sig))
+ return -EINVAL;
+ dilithium_print_polyveck(&ws->buf.h, "Siggen - H K x N matrix:");
+
+ polyveck_use_hint(&ws->w1, &ws->w1, &ws->buf.h);
+ dilithium_print_polyveck(&ws->w1,
+ "Sigver - W K x N matrix after hint:");
+ polyveck_pack_w1(ws->tmp.buf, &ws->w1);
+ dilithium_print_buffer(ws->tmp.buf,
+ DILITHIUM_K * DILITHIUM_POLYW1_PACKEDBYTES,
+ "Sigver - W after w1Encode");
+
+ if (ctx->external_mu) {
+ if (ctx->external_mu_len != DILITHIUM_CRHBYTES)
+ return -EINVAL;
+
+ /* Call random oracle and verify challenge */
+ dilithium_hash_init(ctx);
+ dilithium_hash_update(ctx, ctx->external_mu, DILITHIUM_CRHBYTES);
+ } else {
+ dilithium_hash_final(ctx, ws->buf.mu, DILITHIUM_CRHBYTES);
+
+ /* Call random oracle and verify challenge */
+ dilithium_hash_init(ctx);
+ dilithium_hash_update(ctx, ws->buf.mu, DILITHIUM_CRHBYTES);
+ }
+
+ dilithium_hash_finup(ctx,
+ ws->tmp.buf, DILITHIUM_K * DILITHIUM_POLYW1_PACKEDBYTES,
+ ws->buf.c2.coeffs, DILITHIUM_CTILDE_BYTES);
+
+ /* Signature verification operation */
+ if (memcmp(c1, ws->buf.c2.coeffs, DILITHIUM_CTILDE_BYTES) != 0)
+ ret = -EBADMSG;
+
+out:
+ LC_RELEASE_MEM(ws);
+ return ret;
+}
+
+static int
+dilithium_verify_internal_noahat(const struct dilithium_sig *sig,
+ const struct dilithium_pk *pk,
+ struct dilithium_ctx *ctx)
+{
+ struct workspace_verify {
+ polyvecl mat[DILITHIUM_K];
+ uint8_t poly_uniform_buf[WS_POLY_UNIFORM_BUF_SIZE];
+ };
+ /* The first bytes of the key is rho. */
+ const uint8_t *rho = pk->pk;
+ int ret = 0;
+ LC_DECLARE_MEM(ws, struct workspace_verify, sizeof(uint64_t));
+
+ polyvec_matrix_expand(ws->mat, rho, ws->poly_uniform_buf);
+
+ /* Temporarily set the pointer */
+ ctx->ahat = ws->mat;
+
+ ret = dilithium_verify_internal_ahat(sig, pk, ctx);
+
+ ctx->ahat = NULL;
+ LC_RELEASE_MEM(ws);
+ return ret;
+}
+
+static int dilithium_pk_expand_impl(const struct dilithium_pk *pk,
+ struct dilithium_ctx *ctx)
+{
+ struct workspace_verify {
+ uint8_t poly_uniform_buf[WS_POLY_UNIFORM_BUF_SIZE];
+ };
+ /* The first bytes of the key is rho. */
+ const uint8_t *rho = pk->pk;
+ polyvecl *mat = ctx->ahat;
+ int ret = 0;
+ LC_DECLARE_MEM(ws, struct workspace_verify, sizeof(uint64_t));
+
+ /*
+ * Runtime sanity check ensures that the allocated context has
+ * sufficient size (e.g. not that caller used, say,
+ * DILITHIUM_44_CTX_ON_STACK_AHAT with a ML-DSA 65 or 87 key)
+ */
+#if DILITHIUM_MODE == 2
+ if (ctx->ahat_size < DILITHIUM_44_AHAT_SIZE) {
+ ret = -EOVERFLOW;
+ goto out;
+ }
+#elif DILITHIUM_MODE == 3
+ if (ctx->ahat_size < DILITHIUM_65_AHAT_SIZE) {
+ ret = -EOVERFLOW;
+ goto out;
+ }
+#elif DILITHIUM_MODE == 5
+ if (ctx->ahat_size < DILITHIUM_87_AHAT_SIZE) {
+ ret = -EOVERFLOW;
+ goto out;
+ }
+#else
+#error "Undefined DILITHIUM_MODE"
+#endif
+
+ polyvec_matrix_expand(mat, rho, ws->poly_uniform_buf);
+ ctx->ahat_expanded = 1;
+
+out:
+ LC_RELEASE_MEM(ws);
+ return ret;
+}
+
+static int dilithium_verify_internal(const struct dilithium_sig *sig,
+ const struct dilithium_pk *pk,
+ struct dilithium_ctx *ctx)
+{
+ int ret;
+
+ if (!ctx->ahat)
+ return dilithium_verify_internal_noahat(sig, pk, ctx);
+
+ if (!ctx->ahat_expanded) {
+ ret = dilithium_pk_expand_impl(pk, ctx);
+ if (ret < 0)
+ goto out;
+ }
+
+ ret = dilithium_verify_internal_ahat(sig, pk, ctx);
+
+out:
+ return ret;
+}
+
+static int dilithium_verify_ctx_impl(const struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx,
+ const uint8_t *m, size_t mlen,
+ const struct dilithium_pk *pk)
+{
+ uint8_t tr[DILITHIUM_TRBYTES];
+ int ret = 0;
+
+ if (!sig || !pk || !ctx)
+ return -EINVAL;
+
+ /* Either the message or the external mu must be provided */
+ if (!m && !ctx->external_mu)
+ return -EINVAL;
+
+ /* Make sure that ->mu is large enough for ->tr */
+ BUILD_BUG_ON(DILITHIUM_TRBYTES > DILITHIUM_CRHBYTES);
+
+ /* Compute CRH(H(rho, t1), msg) */
+ shake256(pk->pk, DILITHIUM_PUBLICKEYBYTES, tr,
+ DILITHIUM_TRBYTES);
+
+ if (m) {
+ dilithium_hash_init(ctx);
+ dilithium_hash_update(ctx, tr, DILITHIUM_TRBYTES);
+ ret = signature_domain_separation(
+ &ctx->dilithium_hash_ctx, ctx->ml_dsa_internal,
+ ctx->userctx, ctx->userctxlen,
+ m, mlen,
+ ctx->randomizer, ctx->randomizerlen,
+ DILITHIUM_NIST_CATEGORY);
+ if (ret < 0)
+ goto out;
+ }
+
+ ret = dilithium_verify_internal(sig, pk, ctx);
+
+out:
+ memzero_explicit(tr, sizeof(tr));
+ return ret;
+}
+
+static int dilithium_verify_impl(const struct dilithium_sig *sig,
+ const uint8_t *m, size_t mlen,
+ const struct dilithium_pk *pk)
+{
+ struct dilithium_ctx *ctx;
+ int ret;
+
+ ctx = dilithium_ctx_alloc();
+ if (IS_ERR(ctx))
+ return PTR_ERR(ctx);
+
+ ret = dilithium_verify_ctx_impl(sig, ctx, m, mlen, pk);
+
+ dilithium_ctx_zero_free(ctx);
+ return ret;
+}
+
+static int dilithium_verify_init_impl(struct dilithium_ctx *ctx,
+ const struct dilithium_pk *pk)
+{
+ uint8_t mu[DILITHIUM_TRBYTES];
+
+ /* rng_ctx is allowed to be NULL as handled below */
+ if (!ctx || !pk)
+ return -EINVAL;
+
+ /* Require the use of SHAKE256 */
+ if (!dilithium_hash_check_blocksize(ctx, SHAKE256_BLOCK_SIZE))
+ return -EOPNOTSUPP;
+
+ /* Compute CRH(H(rho, t1), msg) */
+ shake256(pk->pk, DILITHIUM_PUBLICKEYBYTES, mu,
+ DILITHIUM_TRBYTES);
+
+ dilithium_hash_init(ctx);
+ dilithium_hash_update(ctx, mu, DILITHIUM_TRBYTES);
+ memzero_explicit(mu, sizeof(mu));
+
+ return signature_domain_separation(
+ &ctx->dilithium_hash_ctx, ctx->ml_dsa_internal,
+ ctx->userctx, ctx->userctxlen,
+ NULL, 0,
+ ctx->randomizer, ctx->randomizerlen,
+ DILITHIUM_NIST_CATEGORY);
+}
+
+static int dilithium_verify_update_impl(struct dilithium_ctx *ctx,
+ const uint8_t *m, size_t mlen)
+{
+ if (!ctx || !m)
+ return -EINVAL;
+
+ /* Compute CRH(H(rho, t1), msg) */
+ dilithium_hash_update(ctx, m, mlen);
+
+ return 0;
+}
+
+static int dilithium_verify_final_impl(const struct dilithium_sig *sig,
+ struct dilithium_ctx *ctx,
+ const struct dilithium_pk *pk)
+{
+ int ret = 0;
+
+ if (!sig || !ctx || !pk) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ ret = dilithium_verify_internal(sig, pk, ctx);
+
+out:
+ dilithium_ctx_zero(ctx);
+ return ret;
+}
+
+#endif /* DILITHIUM_SIGNATURE_IMPL_H */
diff --git a/crypto/ml_dsa/dilithium_type.h b/crypto/ml_dsa/dilithium_type.h
new file mode 100644
index 000000000000..f9f7ffa2cd38
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_type.h
@@ -0,0 +1,259 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2024 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+
+#ifndef DILITHIUM_TYPE_H
+#define DILITHIUM_TYPE_H
+
+#include "dilithium.h"
+
+/*
+ * This define replaces all symbol names accordingly to allow double compilation
+ * of the same code base.
+ *
+ * Due to the replacement operation, this header file must be included as the
+ * first header file in the entire stack.
+ *
+ * This file can easily be replaced with dilithium.h to achieve the common
+ * functionality without symbol duplication. But in this case, only the
+ * Dilithium security strength is compiled defined in dilithium.h. Duplicate
+ * compilation different sizes would not be possible.
+ */
+#ifdef DILITHIUM_TYPE_65
+#define DILITHIUM_F(name) dilithium_65_##name
+#define dilithium_pk dilithium_65_pk
+#define dilithium_sk dilithium_65_sk
+#define dilithium_sig dilithium_65_sig
+
+#include "dilithium_65.h"
+
+#elif defined DILITHIUM_TYPE_44
+#define DILITHIUM_F(name) dilithium_44_##name
+#define dilithium_pk dilithium_44_pk
+#define dilithium_sk dilithium_44_sk
+#define dilithium_sig dilithium_44_sig
+
+#include "dilithium_44.h"
+
+#else
+#define DILITHIUM_F(name) dilithium_87_##name
+#define dilithium_pk dilithium_87_pk
+#define dilithium_sk dilithium_87_sk
+#define dilithium_sig dilithium_87_sig
+
+#include "dilithium_87.h"
+
+#endif
+
+/*
+ * The following defines simply allow duplicate compilation of the
+ * respective functions.
+ */
+#define dilithium_sign DILITHIUM_F(sign)
+#define dilithium_sign_ctx DILITHIUM_F(sign_ctx)
+#define dilithium_sign_init DILITHIUM_F(sign_init)
+#define dilithium_sign_update DILITHIUM_F(sign_update)
+#define dilithium_sign_final DILITHIUM_F(sign_final)
+#define dilithium_verify DILITHIUM_F(verify)
+#define dilithium_verify_ctx DILITHIUM_F(verify_ctx)
+#define dilithium_verify_init DILITHIUM_F(verify_init)
+#define dilithium_verify_update DILITHIUM_F(verify_update)
+#define dilithium_verify_final DILITHIUM_F(verify_final)
+
+#define dilithium_sign_c DILITHIUM_F(sign_c)
+#define dilithium_sign_ctx_c DILITHIUM_F(sign_ctx_c)
+#define dilithium_sign_init_c DILITHIUM_F(sign_init_c)
+#define dilithium_sign_update_c DILITHIUM_F(sign_update_c)
+#define dilithium_sign_final_c DILITHIUM_F(sign_final_c)
+#define dilithium_verify_c DILITHIUM_F(verify_c)
+#define dilithium_verify_ctx_c DILITHIUM_F(verify_ctx_c)
+#define dilithium_verify_init_c DILITHIUM_F(verify_init_c)
+#define dilithium_verify_update_c DILITHIUM_F(verify_update_c)
+#define dilithium_verify_final_c DILITHIUM_F(verify_final_c)
+
+#define ntt DILITHIUM_F(ntt)
+#define invntt_tomont DILITHIUM_F(invntt_tomont)
+#define poly_chknorm DILITHIUM_F(poly_chknorm)
+#define poly_uniform DILITHIUM_F(poly_uniform)
+#define poly_uniform_eta DILITHIUM_F(poly_uniform_eta)
+#define poly_uniform_gamma1 DILITHIUM_F(poly_uniform_gamma1)
+#define polyz_unpack DILITHIUM_F(polyz_unpack)
+#define poly_challenge DILITHIUM_F(poly_challenge)
+#define polyeta_pack DILITHIUM_F(polyeta_pack)
+#define polyeta_unpack DILITHIUM_F(polyeta_unpack)
+#define polyt1_pack DILITHIUM_F(polyt1_pack)
+#define polyt0_pack DILITHIUM_F(polyt0_pack)
+#define polyt0_unpack DILITHIUM_F(polyt0_unpack)
+#define polyz_pack DILITHIUM_F(polyz_pack)
+#define polyw1_pack DILITHIUM_F(polyw1_pack)
+#define power2round DILITHIUM_F(power2round)
+#define decompose DILITHIUM_F(decompose)
+#define make_hint DILITHIUM_F(make_hint)
+#define use_hint DILITHIUM_F(use_hint)
+
+#define dilithium_print_buffer DILITHIUM_F(print_buffer)
+#define dilithium_print_polyvecl_k DILITHIUM_F(print_polyvecl_k)
+#define dilithium_print_polyvecl DILITHIUM_F(print_polyvecl)
+#define dilithium_print_polyveck DILITHIUM_F(print_polyveck)
+#define dilithium_print_poly DILITHIUM_F(print_poly)
+
+/* AVX2 Implementation */
+#define dilithium_invntt_avx DILITHIUM_F(invntt_avx)
+#define dilithium_ntt_avx DILITHIUM_F(ntt_avx)
+#define dilithium_nttunpack_avx DILITHIUM_F(nttunpack_avx)
+#define dilithium_pointwise_avx DILITHIUM_F(pointwise_avx)
+#define dilithium_pointwise_acc_avx DILITHIUM_F(pointwise_acc_avx)
+#define poly_reduce_avx DILITHIUM_F(poly_reduce_avx)
+#define poly_caddq_avx DILITHIUM_F(poly_caddq_avx)
+#define poly_add_avx DILITHIUM_F(poly_add_avx)
+#define poly_sub_avx DILITHIUM_F(poly_sub_avx)
+#define poly_shiftl_avx DILITHIUM_F(poly_shiftl_avx)
+#define poly_chknorm_avx DILITHIUM_F(poly_chknorm_avx)
+#define poly_uniform_4x_avx DILITHIUM_F(poly_uniform_4x_avx)
+#define poly_uniform_eta_4x_avx DILITHIUM_F(poly_uniform_eta_4x_avx)
+#define poly_uniform_gamma1_4x_avx DILITHIUM_F(poly_uniform_gamma1_4x_avx)
+#define polyz_unpack_avx DILITHIUM_F(polyz_unpack_avx)
+#define poly_challenge_avx DILITHIUM_F(poly_challenge_avx)
+#define polyeta_pack_avx DILITHIUM_F(polyeta_pack_avx)
+#define polyeta_unpack_avx DILITHIUM_F(polyeta_unpack_avx)
+#define polyt1_pack_avx DILITHIUM_F(polyt1_pack_avx)
+#define polyt1_unpack_avx DILITHIUM_F(polyt1_unpack_avx)
+#define polyt0_pack_avx DILITHIUM_F(polyt0_pack_avx)
+#define polyt0_unpack_avx DILITHIUM_F(polyt0_unpack_avx)
+#define polyz_pack_avx DILITHIUM_F(polyz_pack_avx)
+#define polyw1_pack_avx DILITHIUM_F(polyw1_pack_avx)
+#define polyvec_matrix_expand DILITHIUM_F(polyvec_matrix_expand)
+#define polyvec_matrix_expand_row0 DILITHIUM_F(polyvec_matrix_expand_row0)
+#define polyvec_matrix_expand_row1 DILITHIUM_F(polyvec_matrix_expand_row1)
+#define polyvec_matrix_expand_row2 DILITHIUM_F(polyvec_matrix_expand_row2)
+#define polyvec_matrix_expand_row3 DILITHIUM_F(polyvec_matrix_expand_row3)
+#define polyvec_matrix_expand_row4 DILITHIUM_F(polyvec_matrix_expand_row4)
+#define polyvec_matrix_expand_row5 DILITHIUM_F(polyvec_matrix_expand_row5)
+#define polyvec_matrix_expand_row6 DILITHIUM_F(polyvec_matrix_expand_row6)
+#define polyvec_matrix_expand_row7 DILITHIUM_F(polyvec_matrix_expand_row7)
+#define rej_uniform_avx DILITHIUM_F(rej_uniform_avx)
+#define rej_eta_avx DILITHIUM_F(rej_eta_avx)
+#define idxlut DILITHIUM_F(idxlut)
+#define power2round_avx DILITHIUM_F(power2round_avx)
+#define decompose_avx DILITHIUM_F(decompose_avx)
+#define make_hint_avx DILITHIUM_F(make_hint_avx)
+#define use_hint_avx DILITHIUM_F(use_hint_avx)
+#define dilithium_sign_avx2 DILITHIUM_F(sign_avx2)
+#define dilithium_sign_ctx_avx2 DILITHIUM_F(sign_ctx_avx2)
+#define dilithium_sign_init_avx2 DILITHIUM_F(sign_init_avx2)
+#define dilithium_sign_update_avx2 DILITHIUM_F(sign_update_avx2)
+#define dilithium_sign_final_avx2 DILITHIUM_F(sign_final_avx2)
+#define dilithium_verify_avx2 DILITHIUM_F(verify_avx2)
+#define dilithium_verify_ctx_avx2 DILITHIUM_F(verify_ctx_avx2)
+#define dilithium_verify_init_avx2 DILITHIUM_F(verify_init_avx2)
+#define dilithium_verify_update_avx2 DILITHIUM_F(verify_update_avx2)
+#define dilithium_verify_final_avx2 DILITHIUM_F(verify_final_avx2)
+
+/* ARMv8 Implementation */
+#define intt_SIMD_top_armv8 DILITHIUM_F(intt_SIMD_top_armv8)
+#define intt_SIMD_bot_armv8 DILITHIUM_F(intt_SIMD_bot_armv8)
+#define ntt_SIMD_top_armv8 DILITHIUM_F(ntt_SIMD_top_armv8)
+#define ntt_SIMD_bot_armv8 DILITHIUM_F(ntt_SIMD_bot_armv8)
+#define poly_uniformx2 DILITHIUM_F(poly_uniformx2)
+#define poly_uniform_etax2 DILITHIUM_F(poly_uniform_etax2)
+#define poly_uniform_gamma1x2 DILITHIUM_F(poly_uniform_gamma1x2)
+#define armv8_10_to_32 DILITHIUM_F(armv8_10_to_32)
+#define poly_reduce_armv8 DILITHIUM_F(poly_reduce_armv8)
+#define poly_caddq_armv8 DILITHIUM_F(poly_caddq_armv8)
+#define poly_power2round_armv8 DILITHIUM_F(poly_power2round_armv8)
+#define poly_pointwise_montgomery_armv8 \
+ DILITHIUM_F(poly_pointwise_montgomery_armv8)
+#define polyvecl_pointwise_acc_montgomery_armv8 \
+ DILITHIUM_F(polyvecl_pointwise_acc_montgomery_armv8)
+#define dilithium_sign_armv8 DILITHIUM_F(sign_armv8)
+#define dilithium_sign_ctx_armv8 DILITHIUM_F(sign_ctx_armv8)
+#define dilithium_sign_init_armv8 DILITHIUM_F(sign_init_armv8)
+#define dilithium_sign_update_armv8 DILITHIUM_F(sign_update_armv8)
+#define dilithium_sign_final_armv8 DILITHIUM_F(sign_final_armv8)
+#define dilithium_verify_armv8 DILITHIUM_F(verify_armv8)
+#define dilithium_verify_ctx_armv8 DILITHIUM_F(verify_ctx_armv8)
+#define dilithium_verify_init_armv8 DILITHIUM_F(verify_init_armv8)
+#define dilithium_verify_update_armv8 DILITHIUM_F(verify_update_armv8)
+#define dilithium_verify_final_armv8 DILITHIUM_F(verify_final_armv8)
+
+/* ARMv7 Implementation */
+#define armv7_ntt_asm_smull DILITHIUM_F(armv7_ntt_asm_smull)
+#define armv7_inv_ntt_asm_smull DILITHIUM_F(armv7_inv_ntt_asm_smull)
+#define armv7_poly_pointwise_invmontgomery_asm_smull \
+ DILITHIUM_F(armv7_poly_pointwise_invmontgomery_asm_smull)
+#define armv7_poly_pointwise_acc_invmontgomery_asm_smull \
+ DILITHIUM_F(armv7_poly_pointwise_acc_invmontgomery_asm_smull)
+#define poly_uniform_armv7 DILITHIUM_F(poly_uniform_armv7)
+#define armv7_poly_reduce_asm DILITHIUM_F(armv7_poly_reduce_asm)
+#define armv7_rej_uniform_asm DILITHIUM_F(armv7_rej_uniform_asm)
+#define dilithium_sign_armv7 DILITHIUM_F(sign_armv7)
+#define dilithium_sign_ctx_armv7 DILITHIUM_F(sign_ctx_armv7)
+#define dilithium_sign_init_armv7 DILITHIUM_F(sign_init_armv7)
+#define dilithium_sign_update_armv7 DILITHIUM_F(sign_update_armv7)
+#define dilithium_sign_final_armv7 DILITHIUM_F(sign_final_armv7)
+#define dilithium_verify_armv7 DILITHIUM_F(verify_armv7)
+#define dilithium_verify_ctx_armv7 DILITHIUM_F(verify_ctx_armv7)
+#define dilithium_verify_init_armv7 DILITHIUM_F(verify_init_armv7)
+#define dilithium_verify_update_armv7 DILITHIUM_F(verify_update_armv7)
+#define dilithium_verify_final_armv7 DILITHIUM_F(verify_final_armv7)
+
+/* RISCV 64 ASM Implementation */
+#define dilithium_sign_riscv64 DILITHIUM_F(sign_riscv64)
+#define dilithium_sign_ctx_riscv64 DILITHIUM_F(sign_ctx_riscv64)
+#define dilithium_sign_init_riscv64 DILITHIUM_F(sign_init_riscv64)
+#define dilithium_sign_update_riscv64 DILITHIUM_F(sign_update_riscv64)
+#define dilithium_sign_final_riscv64 DILITHIUM_F(sign_final_riscv64)
+#define dilithium_verify_riscv64 DILITHIUM_F(verify_riscv64)
+#define dilithium_verify_ctx_riscv64 DILITHIUM_F(verify_ctx_riscv64)
+#define dilithium_verify_init_riscv64 DILITHIUM_F(verify_init_riscv64)
+#define dilithium_verify_update_riscv64 DILITHIUM_F(verify_update_riscv64)
+#define dilithium_verify_final_riscv64 DILITHIUM_F(verify_final_riscv64)
+#define dilithium_ntt_8l_rv64im DILITHIUM_F(ntt_8l_rv64im)
+#define dilithium_intt_8l_rv64im DILITHIUM_F(intt_8l_rv64im)
+#define dilithium_poly_basemul_8l_init_rv64im \
+ DILITHIUM_F(poly_basemul_8l_init_rv64im)
+#define dilithium_poly_basemul_8l_acc_rv64im \
+ DILITHIUM_F(poly_basemul_8l_acc_rv64im)
+#define dilithium_poly_basemul_8l_acc_end_rv64im \
+ DILITHIUM_F(poly_basemul_8l_acc_end_rv64im)
+#define dilithium_poly_basemul_8l_rv64im DILITHIUM_F(poly_basemul_8l_rv64im)
+#define dilithium_poly_reduce_rv64im DILITHIUM_F(poly_reduce_rv64im)
+
+/* RISCV 64 RVV Implementation */
+#define dilithium_sign_riscv64_rvv DILITHIUM_F(sign_riscv64_rvv)
+#define dilithium_sign_ctx_riscv64_rvv DILITHIUM_F(sign_ctx_riscv64_rvv)
+#define dilithium_sign_init_riscv64_rvv DILITHIUM_F(sign_init_riscv64_rvv)
+#define dilithium_sign_update_riscv64_rvv \
+ DILITHIUM_F(sign_update_riscv64_rvv)
+#define dilithium_sign_final_riscv64_rvv DILITHIUM_F(sign_final_riscv64_rvv)
+#define dilithium_verify_riscv64_rvv DILITHIUM_F(verify_riscv64_rvv)
+#define dilithium_verify_ctx_riscv64_rvv DILITHIUM_F(verify_ctx_riscv64_rvv)
+#define dilithium_verify_init_riscv64_rvv \
+ DILITHIUM_F(verify_init_riscv64_rvv)
+#define dilithium_verify_update_riscv64_rvv \
+ DILITHIUM_F(verify_update_riscv64_rvv)
+#define dilithium_verify_final_riscv64_rvv \
+ DILITHIUM_F(verify_final_riscv64_rvv)
+#define dilithium_ntt_8l_rvv DILITHIUM_F(ntt_8l_rvv)
+#define dilithium_intt_8l_rvv DILITHIUM_F(intt_8l_rvv)
+#define dilithium_poly_basemul_8l_rvv DILITHIUM_F(poly_basemul_8l_rvv)
+#define dilithium_poly_basemul_acc_8l_rvv DILITHIUM_F(poly_basemul_acc_8l_rvv)
+#define dilithium_ntt2normal_order_8l_rvv DILITHIUM_F(ntt2normal_order_8l_rvv)
+#define dilithium_normal2ntt_order_8l_rvv DILITHIUM_F(normal2ntt_order_8l_rvv)
+#define dilithium_poly_reduce_rvv DILITHIUM_F(poly_reduce_rvv)
+
+#endif /* DILITHIUM_TYPE_H */
diff --git a/crypto/ml_dsa/dilithium_zetas.c b/crypto/ml_dsa/dilithium_zetas.c
new file mode 100644
index 000000000000..f0e9203f35dd
--- /dev/null
+++ b/crypto/ml_dsa/dilithium_zetas.c
@@ -0,0 +1,67 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+/*
+ * This code is derived in parts from the code distribution provided with
+ * https://github.com/pq-crystals/dilithium
+ *
+ * That code is released under Public Domain
+ * (https://creativecommons.org/share-your-work/public-domain/cc0/);
+ * or Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0.html).
+ */
+
+#include "dilithium.h"
+
+const int32_t dilithium_zetas[DILITHIUM_N] = {
+ 0, 25847, -2608894, -518909, 237124, -777960, -876248,
+ 466468, 1826347, 2353451, -359251, -2091905, 3119733, -2884855,
+ 3111497, 2680103, 2725464, 1024112, -1079900, 3585928, -549488,
+ -1119584, 2619752, -2108549, -2118186, -3859737, -1399561, -3277672,
+ 1757237, -19422, 4010497, 280005, 2706023, 95776, 3077325,
+ 3530437, -1661693, -3592148, -2537516, 3915439, -3861115, -3043716,
+ 3574422, -2867647, 3539968, -300467, 2348700, -539299, -1699267,
+ -1643818, 3505694, -3821735, 3507263, -2140649, -1600420, 3699596,
+ 811944, 531354, 954230, 3881043, 3900724, -2556880, 2071892,
+ -2797779, -3930395, -1528703, -3677745, -3041255, -1452451, 3475950,
+ 2176455, -1585221, -1257611, 1939314, -4083598, -1000202, -3190144,
+ -3157330, -3632928, 126922, 3412210, -983419, 2147896, 2715295,
+ -2967645, -3693493, -411027, -2477047, -671102, -1228525, -22981,
+ -1308169, -381987, 1349076, 1852771, -1430430, -3343383, 264944,
+ 508951, 3097992, 44288, -1100098, 904516, 3958618, -3724342,
+ -8578, 1653064, -3249728, 2389356, -210977, 759969, -1316856,
+ 189548, -3553272, 3159746, -1851402, -2409325, -177440, 1315589,
+ 1341330, 1285669, -1584928, -812732, -1439742, -3019102, -3881060,
+ -3628969, 3839961, 2091667, 3407706, 2316500, 3817976, -3342478,
+ 2244091, -2446433, -3562462, 266997, 2434439, -1235728, 3513181,
+ -3520352, -3759364, -1197226, -3193378, 900702, 1859098, 909542,
+ 819034, 495491, -1613174, -43260, -522500, -655327, -3122442,
+ 2031748, 3207046, -3556995, -525098, -768622, -3595838, 342297,
+ 286988, -2437823, 4108315, 3437287, -3342277, 1735879, 203044,
+ 2842341, 2691481, -2590150, 1265009, 4055324, 1247620, 2486353,
+ 1595974, -3767016, 1250494, 2635921, -3548272, -2994039, 1869119,
+ 1903435, -1050970, -1333058, 1237275, -3318210, -1430225, -451100,
+ 1312455, 3306115, -1962642, -1279661, 1917081, -2546312, -1374803,
+ 1500165, 777191, 2235880, 3406031, -542412, -2831860, -1671176,
+ -1846953, -2584293, -3724270, 594136, -3776993, -2013608, 2432395,
+ 2454455, -164721, 1957272, 3369112, 185531, -1207385, -3183426,
+ 162844, 1616392, 3014001, 810149, 1652634, -3694233, -1799107,
+ -3038916, 3523897, 3866901, 269760, 2213111, -975884, 1717735,
+ 472078, -426683, 1723600, -1803090, 1910376, -1667432, -1104333,
+ -260646, -3833893, -2939036, -2235985, -420899, -2286327, 183443,
+ -976891, 1612842, -3545687, -554416, 3919660, -48306, -1362209,
+ 3937738, 1400424, -846154, 1976782
+};
diff --git a/crypto/ml_dsa/fips_mode.h b/crypto/ml_dsa/fips_mode.h
new file mode 100644
index 000000000000..f185f204366e
--- /dev/null
+++ b/crypto/ml_dsa/fips_mode.h
@@ -0,0 +1,45 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+
+#ifndef FIPS_MODE_H
+#define FIPS_MODE_H
+
+/**
+ * @brief Is FIPS 140 Mode enabled?
+ *
+ * return 0 == false, 1 == true
+ */
+static inline bool fips140_mode_enabled(void)
+{
+ return false;
+}
+
+#define FIPS140_PCT_LOOP(func) \
+ if (fips140_mode_enabled()) { \
+ unsigned int __i; \
+ int __ret; \
+ \
+ for (__i = 0; __i < 5; __i++) { \
+ __ret = func; \
+ if (!__ret) \
+ return __ret; \
+ } \
+ WARN_ON(0); \
+ }
+
+#endif /* FIPS_MODE_H */
diff --git a/crypto/ml_dsa/signature_domain_separation.c b/crypto/ml_dsa/signature_domain_separation.c
new file mode 100644
index 000000000000..f0a173797c96
--- /dev/null
+++ b/crypto/ml_dsa/signature_domain_separation.c
@@ -0,0 +1,213 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * Copyright (C) 2024 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+
+#include <crypto/sha3.h>
+#include <crypto/hash.h>
+#include "signature_domain_separation.h"
+
+static const char *signature_prehash_type;
+
+/* RFC4055 2.16.840.1.101.3.4.2.1 */
+static const uint8_t sha256_oid_der[] __maybe_unused = { 0x06, 0x09, 0x60, 0x86,
+ 0x48, 0x01, 0x65, 0x03,
+ 0x04, 0x02, 0x01 };
+/* RFC4055 2.16.840.1.101.3.4.2.2 */
+static const uint8_t sha384_oid_der[] __maybe_unused = { 0x06, 0x09, 0x60, 0x86,
+ 0x48, 0x01, 0x65, 0x03,
+ 0x04, 0x02, 0x02 };
+/* RFC4055 2.16.840.1.101.3.4.2.3 */
+static const uint8_t sha512_oid_der[] __maybe_unused = { 0x06, 0x09, 0x60, 0x86,
+ 0x48, 0x01, 0x65, 0x03,
+ 0x04, 0x02, 0x03 };
+
+/*
+ * https://lamps-wg.github.io/draft-composite-sigs/draft-ietf-lamps-pq-composite-sigs.html
+ */
+static const uint8_t sha3_256_oid_der[] __maybe_unused = {
+ 0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x08
+};
+static const uint8_t sha3_384_oid_der[] __maybe_unused = {
+ 0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x09
+};
+static const uint8_t sha3_512_oid_der[] __maybe_unused = {
+ 0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x0a
+};
+
+/* RFC8692 2.16.840.1.101.3.4.2.11 */
+static const uint8_t shake128_oid_der[] __maybe_unused = {
+ 0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x0B
+};
+
+/* RFC8692 2.16.840.1.101.3.4.2.11 */
+static const uint8_t shake256_oid_der[] __maybe_unused = {
+ 0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x0C
+};
+
+int signature_ph_oids(struct shake256_ctx *hash_ctx, size_t mlen,
+ unsigned int nist_category)
+{
+ /* If no hash is supplied, we have no HashML-DSA */
+ if (!signature_prehash_type)
+ return 0;
+
+ /*
+ * The signature init/update/final operation will not work with the
+ * check of mlen, as only when _final is invoked, the message length
+ * is known.
+ *
+ * As defined in FIPS 204, section 5.4 requires
+ * "... the digest that is signed needs to be generated using an
+ * approved hash function or XOF (e.g., from FIPS 180 or FIPS 202) that
+ * provides at least λ bits of classical security strength against both
+ * collision and second preimage attacks ... Obtaining at least λ bits
+ * of classical security strength against collision attacks requires
+ * that the digest to be signed be at least 2λ bits in length."
+ * This requirement implies in the following definitions.
+ */
+ (void)mlen;
+
+ switch (nist_category) {
+ case 1:
+ if (strcmp(signature_prehash_type, "sha256") == 0) {
+ // if (mlen != LC_SHA256_SIZE_DIGEST)
+ // return -EOPNOTSUPP;
+ shake256_update(hash_ctx, sha256_oid_der,
+ sizeof(sha256_oid_der));
+ return 0;
+ }
+ if (strcmp(signature_prehash_type, "sha3-256") == 0) {
+ // if (mlen != LC_SHA3_256_SIZE_DIGEST)
+ // return -EOPNOTSUPP;
+ shake256_update(hash_ctx, sha3_256_oid_der,
+ sizeof(sha3_256_oid_der));
+ return 0;
+ }
+ if (strcmp(signature_prehash_type, "shake128") == 0) {
+ /* FIPS 204 section 5.4.1 */
+ // if (mlen != 32)
+ // return -EOPNOTSUPP;
+ shake256_update(hash_ctx, shake128_oid_der,
+ sizeof(shake128_oid_der));
+ return 0;
+ }
+ /* FALLTHROUGH - Dilithium44 allows the following, too */
+ fallthrough;
+ case 3:
+ if (strcmp(signature_prehash_type, "sha3-384") == 0) {
+ // if (mlen != LC_SHA3_384_SIZE_DIGEST)
+ // return -EOPNOTSUPP;
+ shake256_update(hash_ctx, sha3_384_oid_der,
+ sizeof(sha3_384_oid_der));
+ return 0;
+ }
+ if (strcmp(signature_prehash_type, "sha384") == 0) {
+ // if (mlen != LC_SHA384_SIZE_DIGEST)
+ // return -EOPNOTSUPP;
+ shake256_update(hash_ctx, sha384_oid_der,
+ sizeof(sha384_oid_der));
+ return 0;
+ }
+ /* FALLTHROUGH - Dilithium[44|65] allows the following, too */
+ fallthrough;
+ case 5:
+ if (strcmp(signature_prehash_type, "sha512") == 0) {
+ // if (mlen != LC_SHA512_SIZE_DIGEST)
+ // return -EOPNOTSUPP;
+ shake256_update(hash_ctx, sha512_oid_der,
+ sizeof(sha512_oid_der));
+ return 0;
+ }
+ if (strcmp(signature_prehash_type, "sha3-512") == 0) {
+ // if (mlen != LC_SHA3_512_SIZE_DIGEST)
+ // return -EOPNOTSUPP;
+ shake256_update(hash_ctx, sha3_512_oid_der,
+ sizeof(sha3_512_oid_der));
+ return 0;
+ } else if (strcmp(signature_prehash_type, "shake256") == 0) {
+ /* FIPS 204 section 5.4.1 */
+ /*
+ * TODO: mlen must be >= 64 to comply with the
+ * aforementioned requirement - unfortunately we can
+ * only check mlen at the end of the signature
+ * operation - shall this be implemented?
+ */
+ // if (mlen != 64)
+ // return -EOPNOTSUPP;
+ shake256_update(hash_ctx, shake256_oid_der,
+ sizeof(shake256_oid_der));
+ return 0;
+ }
+ break;
+ default:
+ break;
+ }
+
+ return -EOPNOTSUPP;
+}
+
+/* FIPS 204 pre-hash ML-DSA domain separation, but without original message */
+static int standalone_signature_domain_separation(
+ struct shake256_ctx *hash_ctx, const uint8_t *userctx,
+ size_t userctxlen, size_t mlen, unsigned int nist_category)
+{
+ uint8_t domainseparation[2];
+
+ domainseparation[0] = signature_prehash_type ? 1 : 0;
+ domainseparation[1] = (uint8_t)userctxlen;
+
+ shake256_update(hash_ctx, domainseparation, sizeof(domainseparation));
+ shake256_update(hash_ctx, userctx, userctxlen);
+
+ return signature_ph_oids(hash_ctx, mlen, nist_category);
+}
+
+/*
+ * Domain separation as required by:
+ *
+ * FIPS 204 pre-hash ML-DSA: randomizer is NULL
+ * Composite ML-DSA draft 5: randomizer is set
+ */
+int signature_domain_separation(struct shake256_ctx *hash_ctx,
+ unsigned int ml_dsa_internal,
+ const uint8_t *userctx, size_t userctxlen,
+ const uint8_t *m, size_t mlen,
+ const uint8_t *randomizer, size_t randomizerlen,
+ unsigned int nist_category)
+{
+ int ret = 0;
+
+ /* The internal operation skips the domain separation code */
+ if (ml_dsa_internal)
+ goto out;
+
+ if (userctxlen > 255)
+ return -EINVAL;
+
+ /* If Composite ML-DSA is requested, use domain as userctx */
+ if (randomizer) {
+ return -EOPNOTSUPP;
+ } else {
+ ret = standalone_signature_domain_separation(
+ hash_ctx, userctx, userctxlen,
+ mlen, nist_category);
+ }
+
+out:
+ shake256_update(hash_ctx, m, mlen);
+ return ret;
+}
diff --git a/crypto/ml_dsa/signature_domain_separation.h b/crypto/ml_dsa/signature_domain_separation.h
new file mode 100644
index 000000000000..01dafaa851bf
--- /dev/null
+++ b/crypto/ml_dsa/signature_domain_separation.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2024 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+
+#ifndef SIGNATURE_DOMAIN_SEPARATION_H
+#define SIGNATURE_DOMAIN_SEPARATION_H
+
+#include <crypto/sha3.h>
+
+int signature_domain_separation(struct shake256_ctx *hash_ctx,
+ unsigned int ml_dsa_internal,
+ const uint8_t *userctx, size_t userctxlen,
+ const uint8_t *m, size_t mlen,
+ const uint8_t *randomizer, size_t randomizerlen,
+ unsigned int nist_category);
+int signature_ph_oids(struct shake256_ctx *hash_ctx, size_t mlen,
+ unsigned int nist_category);
+
+#endif /* SIGNATURE_DOMAIN_SEPARATION_H */
diff --git a/crypto/ml_dsa/small_stack_support.h b/crypto/ml_dsa/small_stack_support.h
new file mode 100644
index 000000000000..9c1eba6c40bf
--- /dev/null
+++ b/crypto/ml_dsa/small_stack_support.h
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2022 - 2025, Stephan Mueller <smueller@chronox.de>
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
+ * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+
+#ifndef SMALL_STACK_SUPPORT_H
+#define SMALL_STACK_SUPPORT_H
+
+/* Allocate memory on heap */
+#define __LC_DECLARE_MEM_HEAP(name, type, alignment) \
+ type *name = kzalloc(round_up(sizeof(type), alignment), GFP_KERNEL); \
+ if (!name) \
+ return -ENOMEM; \
+
+#define __LC_RELEASE_MEM_HEAP(name) \
+ kfree_sensitive(name);
+
+#define noinline_stack noinline
+
+#define LC_DECLARE_MEM(name, type, alignment) \
+ _Pragma("GCC diagnostic push") \
+ _Pragma("GCC diagnostic ignored \"-Wcast-align\"") \
+ __LC_DECLARE_MEM_HEAP(name, type, alignment); \
+ _Pragma("GCC diagnostic pop")
+#define LC_RELEASE_MEM(name) __LC_RELEASE_MEM_HEAP(name)
+
+#endif /* SMALL_STACK_SUPPORT_H */
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox