[igt-dev] Scripting language for igt

* [igt-dev] Scripting language for igt
@ 2018-08-20 10:11 Chris Wilson
  2018-08-21 11:43 ` Joonas Lahtinen
  0 siblings, 1 reply; 7+ messages in thread
From: Chris Wilson @ 2018-08-20 10:11 UTC (permalink / raw)
  To: igt-dev

Over the years, I written a few stack-based bytecode interpreters for
use with a fuzzer (afl + kcov) for both cairo and igt. However, the
languages are PostScript-like; elegant in their simplicity but writing
in them by hand is a challenge. They were intended for automated test
generation.

However, with the inclusion of a test runner within igt, we can actually
take that a step further an develop it into a test server with an
interpreter builtin. The major of the runtime for a trivial negative
test (over 90% of say igt/gem_basic) is spent loading the executable
into memory; during that time an igt shell could have run many tests.

Now it would be possible to integrate our C into one huge binary and
have the igt test runner safely fork around each subtest (or subtest
group), or we can look at a scripting language to ease writing tests,
and to ease automated test generation.

lua (and especially luajit) are a compelling choice for an embedded
interpreter, but the language is tricky without a wide array of external
tooling. 

Angelscript is nice if we are looking for a statically typed
interpreted language close to C. But is slow and has even less user base
then lua.

Python? Different to C, which can be both a blessing and aggravating at
times. Slow (incredibly slow) and bloated (incredibly bloated, it
doesn't really mix well with the forkserver strategy as even
copy-on-write is not enough to paper over its memory consumption).
However, it is reasonably familiar with a huge library of tools and
users.

Javascript. Once derided for being worse than php, has grown up into
quite a nice little language. From C-like in ECMAScript 5 to declarative
with lambda promises in ECMAScript 6, it provides a lot of convenience for
writing test cases.

For a change, not having used javascript before, I picked up
https://duktape.org/ over the weekend, as it was described as having a
very easy to integrate lua-like C-interface. (And yes it does on both
accounts). After having a look at embedding v8, it was pretty trivial to
setup.

As a taste of what a javascript newbie can do, I wrote a few equivalents
for our BAT:

igt/core_auth/basic-auth => tests/drm/auth.js
---
const nop_ioctl = function() { this.ioctl(drm.ioctl.DRM_IOCTL_FINISH) }

igt.fork(drm.cards(), function(master) {
        var slave, magic, root, authorized;

        // Confirm we are a master and open a second fd to the same device
        root = true;
        try {
                master.SetMaster();
        } catch(e) {
                // Alas not starting as root, we'll just have to hope.
                igt.assert(os.uid());
                igt.assert(e.errno === -os.EACCES);
                root = false;
        }
        slave = master.reopen();
        authorized = root;

        for (var depth = 0; depth < 3; depth++) {
                if (!authorized) {
                        // If !authorized (and !root) auth-only ioctls will fail
                        igt.expect(function(){ nop_ioctl.call(slave) },
                                   -os.EACCES);
                }

                // Grab the magic, and check it is constant
                magic = slave.GetMagic();
                igt.assert(slave.GetMagic() === magic);

                // A slave cannot simply authorize their own magic
                igt.expect(function(){ slave.AuthMagic(magic) }, -os.EACCES);

                // Magic works first time
                master.AuthMagic(magic);

                // But it can only be authorized once
                igt.expect(function(){ master.AuthMagic(magic) }, -os.EINVAL);

                // Verify the magic did not change
                igt.assert(slave.GetMagic() === magic);

                // And the consumed magic still fails if used again by the slave
                igt.expect(function(){ slave.AuthMagic(magic) }, -os.EACCES);

                // And confirm the slave is now authorized
                nop_ioctl.call(slave);

                // But not a master
                if (root)
                        igt.expect(function(){ slave.SetMaster() }, -os.EINVAL);

                // A second slave is not authorized using the old magic
                slave = slave.reopen();
                igt.expect(function(){ slave.AuthMagic(magic) }, -os.EACCES);
                authorized = root;

                if (root) {
                        os.drop_root();
                        root = false;
                }
        }
})
---
and igt/gem_basic => tests/i915/gem_basic.js:
---
const i915 = igt.require(drm.drivers.i915.card());

const i_create = i915.driver.ioctl.DRM_IOCTL_I915_GEM_CREATE;
const i_open = i915.driver.drm.ioctl.DRM_IOCTL_GEM_OPEN;
const i_flink = i915.driver.drm.ioctl.DRM_IOCTL_GEM_FLINK;
const i_close = i915.driver.drm.ioctl.DRM_IOCTL_GEM_CLOSE;

var arg = new Uint32Array(4);
var handle;

const create = function(size) {
	arg[0] = size; /* only up to 4GiB! */
	arg[1] = 0;
	i915.ioctl(i_create, arg);
	return arg[2];
}

const flink = function(handle) {
	arg[0] = handle;
	i915.ioctl(i_flink, arg);
	return arg[1];
}

const open = function(name) {
	arg[0] = name;
	i915.ioctl(i_open, arg);
	return arg[1];
}

const clone = function(handle) {
	return open(flink(handle));
}

const close = function(handle) {
	arg[0] = handle;
	i915.ioctl(i_close, arg);
}

const bad_close = function(handle) {
	arg[0] = handle;
	i915.expect_ioctl(i_close, arg, -os.EINVAL);
}

const bad_values = [
	0x00000000,
	0x00000001,
	0x00000010,
	0x00000100,
	0x00001000,
	0x00010000,
	0x00100000,
	0x01000000,
	0x10000000,
	0x00010001,
	0x00100010,
	0x01000100,
	0x10001000,
	0x01010101,
	0x10101010,
	0x11111111,
	0x00c00fee,
	0xdeadbeef,
	0xcccccccc,
	0xaaaaaaaa,
	0xa5a5a5a5,
	0xc3c3c3c3,
	0x0000ffff,
	0x0007ffff,
	0x0008ffff,
	0x000fffff,
	0x007fffff,
	0x008fffff,
	0x00ffffff,
	0x07ffffff,
	0x08ffffff,
	0x0fffffff,
	0x7fffffff,
	0x8fffffff,
	0xffffffff,
];

/* Check that all bad values (and their inverse) generate -EINVAL */
bad_values.forEach(function(val) { bad_close(val); bad_close(~val); });

/* Check basics do work after all */
close(create(4096));

/* And that we disallow double closes */
close(handle = create(4096));
bad_close(handle);

/* But cloning does produce two distinct handles */
close(clone(handle = create(4096)));
close(handle);

/* More clones; an army of them. */
handle = create(4096);
const count = 4096;
const handles = new Array(count);
for(var i = 0; i < count; i++)
	handles[i] = clone(handle);
close(handle);
Math.permute(handles).forEach(close);
Math.permute(handles).forEach(bad_close);
bad_close(handle);

/* And now that we're empty again, everything should be bad. */
bad_values.forEach(function(val) { bad_close(val); bad_close(~val); });
---

In both cases it was easy to extend the test beyond the original. Joonas
who is more familiar with the later declarative ECMAScript, says he can
reduce the boilerplate code even further. (Some of the verbosity is on
purpose, since these tests are negative tests and so want to need to dig
beneath the usual layers of convenience api, but still the plan should
be to autogenerate as much of the ioctl breaking as possible.)

TLDR; I want to rewrite BAT and use a script interpreter for speed of
execution, ease of test writing, and for automating test generation. 
I would like to drop duktape into igt/shell, but the initial code drop
may be too big to post/review... So I would like to present it as fait
accompli and then work on the smaller patches to make the magic happen.
-Chris
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

^ permalink raw reply	[flat|nested] 7+ messages in thread