Re: [LTP] [PATCH 0/7] docparse improvements

Richard Palethorpe
2021-10-28T09:54:10Z

Hello,

Cyril Hrubis <chrubis@suse.cz> writes:

> Hi!
>> It's unfortunate that before starting this effort and the checker that
>> we didn't know about tree-sitter (although Sparse may still be the best
>> choice for the checker).
>>
>> Tree-sitter can parse C into an AST and can easily be vendored into LTP:
>> https://tree-sitter.github.io/tree-sitter/using-parsers#building-the-library
>>
>> Then we just need to work on the level of the AST. It also has a query
>> language. This should allow the initial matching to be done on a high
>> level.
>
> The only worry that I have about this would be speed, currently the code
> I wrote takes a few second to process thousands of C files in LTP, that
> is because we take a lot of shortcuts and ignore all the stuff we do not
> need. Full parser that builds AST would be orders of magnitude slower,
> so before we attempt to use it it should be benchmarked properly to see
> if it's fast enough.

It's incredibly fast, it has no trouble parsing the entire kernel.

Weggli uses tree-sitter

https://github.com/googleprojectzero/weggli

rich@g78 ~/q/ltp (master)> time weggli '_ verify_alarm(_) { exit(0); }' .
/home/rich/qa/ltp/./testcases/kernel/syscalls/alarm/alarm03.c:21
static void verify_alarm(void)
{
pid_t pid;

TEST(alarm(100));

..
} else {
tst_res(TPASS,
"alarm(100), fork, alarm(0) child's "
"alarm returned %ld", TST_RET);
}
exit(0);
}

TEST(alarm(0));
if (TST_RET != 100) {
tst_res(TFAIL,
..
}
/home/rich/qa/ltp/./testcases/kernel/syscalls/alarm/alarm07.c:20
static void verify_alarm(void)
{
pid_t pid;
alarm_cnt = 0;

TEST(alarm(1));
..
tst_res(TPASS, "alarm() request cleared in child");
} else {
tst_res(TFAIL, "alarm() request not cleared in "
"child; alarms received:%d", alarm_cnt);
}
exit(0);
}

if (alarm_cnt != 1)
tst_res(TFAIL, "Sigalarms in parent %i, expected 1", alarm_cnt);
else
..
}

________________________________________________________
Executed in 49.35 millis fish external
usr time 110.88 millis 0.00 millis 110.88 millis
sys time 87.44 millis 1.20 millis 86.24 millis

>
>> If we continue down the path of hand parsing C, then it will most likely
>> result in constant tweaks and additions.
>
> Well I would say that this patchset is the last addition for the parser,
> if we ever need anything more complex we should really switch to
> something else. On the other hand I do not think that we will ever need
> more complexity in the parser than this, as long as we keep things
> sane.

This closes the door on a lot of options for no upside AFAICT. We have
two tools (Sparse and tree-sitter) that can be (or have been) vendored
and will parse a large subset of C. Sparse goes a step further allowing
control flow analysis. The usual reasons for reinventing the wheel are
not present.

--
Thank you,
Richard.