kcalloc/kmalloc_array could BUILD_BUG_ON for too-big constant arguments (was Re: [PATCH] [RESEND] aic7xxx: replace kmalloc/memset by kzalloc)

[Jeff Epler <jepler@unpythonic.net>]

The following is a sketch of how a macro kcalloc could BUILD_BUG_ON for
overflows of two compile-time operands, or call "kcalloc_variable" for
nonconstant arguments.  Tested on gcc 4.7.2 only, since it's what I had to
hand.  I didn't do any testing beyond checking that fn2 didn't build, and that
fn1/3 had plausible-looking code on x86_64.

typedef unsigned long size_t;
#define SIZE_MAX (~(size_t)0)
typedef int gfp_t;
extern void *kzalloc(size_t n, gfp_t flags);
extern void *kcalloc_variable(size_t n, size_t size, gfp_t flags);
#define BUILD_BUG_ON(condition) ((void)sizeof(char[1 - 2*!!(condition)]))

#define kcalloc(n, size, flags) \
	__builtin_choose_expr(__builtin_constant_p((n) | (size)), \
		( \
			BUILD_BUG_ON((n) > SIZE_MAX / (size)), \
			kzalloc((n) * (size), (flags)) \
		), kcalloc_variable((n), (size), (flags)))


void fn1() { kcalloc(3, 3, 0); }
//void fn2() { kcalloc(2, ~(size_t)0, 0); }// compile-time BUILD_BUG_ON
void fn3(int i) { kcalloc(2, i, 0); }

Jeff