Re: [RFC PATCH] pre-process: add __VA_OPT__ support

[Al Viro <viro@zeniv.linux.org.uk>]

On Mon, Mar 16, 2026 at 06:56:22AM +0000, Al Viro wrote:

... and here's are the whitespace fixes and a bunch of optimizations; I
really wonder if we ought to simply switch to uint64_t for struct position
contents and go for several helpers for accessing individual parts.
Code generation with clang is reasonable, but gcc is bloody awful -
I'm sick and tired of fighting it all the time ;-/ Anyway, I've spent
way too much time in that rabbit hole.  I've a few more sitting in the
local queue (tokenizer optimizations, mostly), but that's a separate
story and I'd rather leave that for later.

See git://git.kernel.org/pub/scm/linux/kernel/git/viro/sparse.git #whitespaces
(or individual patches in followups); it's on top of #va_opt.

Re whitespace: there are several places where gcc and clang handling of
that stuff differs.  The origin of the headache is that historically
preprocessor had been a text filter sitting in front of compiler -
its output had been fed through tokenizer and quite a few things in the
original ANSI C bear the resulting scars.  By C99 times it mostly switched
to working in terms of token stream, which simplified a lot of stuff.
Unfortunately, what it's working with is not quite a list of tokens -
it's tokens mixed with whitespace.  By that point comments are replaced
with spaces and implementation is allowed to collapse any sequence of
whitespace other than newline into a single space character.

All operations on the stream are described in terms of replacing a
token or a sequence of tokens with a sequence of tokens; what happens
to possible whitespace is almost unspecified.

Some things are obvious and generally agreed upon:
	* if we are replacing a sequence of tokens, all whitespace stuck
between them is removed.  All whitespace before to the first or after the
last affected token in left as-is.
	* all whitespace stuck between the tokens in the replacement
sequence is inserted along with those tokens.

What is *not* obvious is what should be done to leading or trailing
whitespace in the replacement sequence.  Part of that is explicitly
covered (e.g. macro body has no leading or trailing whitespace), but in
some cases the things get iffy.  Trivial example:

#define E
#define F E - E
#define S(X) #X
#define A(X) S([X])
char *s = A(F);

A(F) is a macro invocation; there's one argument - Ident(F).  No whitespace
anywhere.  What should these 4 tokens be replaced with?  We start with the
replacement list of A - Ident(S),Punctuator('('),Punctuator('['),Ident(X),
Punctuator(']'),Punctuator(')').  The only parameter (X) occurs only once
and there's neither # nor ## operator in sight, so that occurrence gets
replaced with the token sequence resulting from full expansion of the
corresponding argument.

Ident(F) expands to Ident(E),whitespace,Punctuator('-'),whitespace,Ident(E),
which, after expanding both instances of E turns into
	whitespace,Punctuator('-'),whitespace

Fair enough, there's only one token in that, so there's no inter-token
whitespace in sight.  There is both leading and trailing whitespace, though.

One interpretation is that this one token is precisely what gets substituted,
yielding  S([-]) with no whitespace anywhere.  That's an invocation of
S with the sole argument being Punctuator('['),Punctuator('-),Punctuator(']'),
and the end result is "[-]".  This is what clang does.

Another way to handle that is to have the whitespace preceding and following
Punctuator('-') in the expansion of F stuck to it when we are substituting
the damn thing, yielding S([ - ]) (same tokens as in the first variant, but
with whitespaces stuck before and after the minus).  The end result in this
interpretation is "[ - ]".  This is what gcc goes for.

Unfortunately, gcc rules are convoluted and when you add __VA_OPT__ into
the mix they get really nasty.

Example:
#define E
#define F E - E
#define S(X) #X
#define A(X,...) S([__VA_OPT__(X)])
char *s = A(F,_);

ends up with "[- ]" - the leading whitespace is stripped, the trailing one
remains stuck to the token sequence.  Mixing that with ## leads to outright
bugs:

#define E
#define F1 E 1
#define A(X,...) 1##__VA_OPT__(X)
int m1 = A(F1,...);

expands into

int m1 = 11;

In other words, left concatenation works fine.  However, adding the right-hand
analogue

#define F2 sizeof m E
#define B(X,...) __VA_OPT__(X)##1
int m2 = B(F2,_);

which should result in

int m2 = sizeof m1;

does *not* work with gcc - it ends up with

int m2 = sizeof m 1;

with predictable objections from the parser.  Note that it's *not* a matter of
confusion re "do we expand the argument here?" - if that argument hadn't been
expanded, we would've ended up with

int m2 = sizeof F21;

and resulting errors would've been different.  With clang everything works
as expected and I don't see any interpretation of the standard that would
allow what gcc is doing.  The trigger for that bug is their treatment of the
trailing whitespace, AFAICS; leading whitespace is stripped here.

Note that the current mainline sparse doesn't match either gcc or clang
semantics - it loses whitespaces in some cases where they definitely should
be retained.  That needs to be fixed; e.g.
#define E
#define S(X) #X
#define A(X) S(X)
char *s = A([1 E]);
ought to produce "[1 ]", not "[1]".  Nothing subtle about that case...

IMO we are be better off with the clang approach; it is easy to describe
and it matches the behaviour of gcc unless you step into the areas where gcc
is rather random.  We can always reconsider if/when gcc folks come up with
something that looks consistent...

Shortlog:

Al Viro (6):
      nextchar(): get rid of special[]
      simplify the inlined side of nextchar()
      tokenize_stream(): don't bother with isspace()
      TOKEN_DIRECTIVE: recognize directive-introducing # in tokenizer
      saner collect_arg() code generation
      try to get whitespaces right

Diffstat:
 pre-process.c                           | 169 ++++++++++++++++++--------------
 token.h                                 |   1 +
 tokenize.c                              |  91 +++++++++--------
 validation/preprocessor/va_opt-concat.c |  24 +++++
 validation/preprocessor/va_opt.c        |  23 +++--
 validation/preprocessor/va_opt2.c       |   2 +-
 6 files changed, 184 insertions(+), 126 deletions(-)
 create mode 100644 validation/preprocessor/va_opt-concat.c