Why don't you just have multiple image recipes?

The initial question was not related to image recipes, but rather to the one linux-kernel recipe. Within the recipe there's no variable I could use to specify which modules get compiled.

Many images bluntly install "kernel-modules" which throws in everything that got built by the kernel recipe. Omit that and you can install kernel modules based on exact system requirements.

This is something I will take a look at, thanks for the idea. One "issue" I see with it is that this won't work for built-in modules, however, those "special" scenarios can be covered with modules loaded on demand.

Aleksandar