[ARM ATTEND] Interested in R/M-class (!MMU), automated testing

[ARM ATTEND] Interested in R/M-class (!MMU), automated testing

[Jonathan Austin]

Hello,

I'm keen to attend the ARM Kernel summit this year. I haven't got a 
specific proposal, but I'd really like to talk to people about !MMU, 
R/M-class (which is where I've done most of my work so far) and 
automation of testing on ARM hardware.

I'm pretty new to the kernel and based on the accounts of other hackers, 
getting to the ARM (mini) summits and meeting people in person is a 
great way to get more involved, so if there are places I'd love to come.

Specifically, after the merging of my MPU support for R-class and 
Uwe/Catalin's M-class patches, I'd like to talk about:

- Whether we can make better use of the MPU/actually have real
   kernel/userspace protection even without an MMU, and how not to kill
   performance doing this
- Whether we want to do MPU support for M-class
- How we can get more eyes on the !MMU code in the kernel - is it
   because nobody's got R/M hardware? R/M isn?t interesting to people so
   far? etc
- Ways to do more automated testing of ARM kernels, including hearing
   from other people what they use to build/boot/test/benchmark their
   code (perhaps even get a discussion going about using the diverse
   range of hardware people have around put to use as test-machines).

Thanks,
Jonny

[ARM ATTEND] Interested in R/M-class (!MMU), automated testing

[Robert Schwebel]

Hi Jonathan,

On Fri, Aug 02, 2013 at 07:23:57PM +0100, Jonathan Austin wrote:
> - How we can get more eyes on the !MMU code in the kernel - is it
>   because nobody's got R/M hardware? R/M isn?t interesting to people so
>   far? etc

Since my team (mainly Uwe) has worked on the Cortex-M port, I still
havn't seen many real usecases where Cortex-M and Linux made sense. The
original idea was to use EFM32 because it was a) very low power and b)
got a high quality network stack with Linux.

Unfortunately, most requests we got since then were from people who
expected an extraordinairy "cheap" system, but that doesn't work if you
look at the BOM for a complete system, both in terms of money and power
consumption.

That may be a reason for having not that many hardware out there.

There are some FPGAs with Cortex-M out there, which might be an
interesting target.

> - Ways to do more automated testing of ARM kernels, including hearing
>   from other people what they use to build/boot/test/benchmark their
>   code (perhaps even get a discussion going about using the diverse
>   range of hardware people have around put to use as test-machines).

That's pretty interesting; we have a test farm at PTX, but it mainly
does nightly build + boot tests on mainline, for systems we care of.

rsc
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

[ARM ATTEND] Interested in R/M-class (!MMU), automated testing

[Olof Johansson]

On Sun, Aug 4, 2013 at 5:25 AM, Robert Schwebel
<r.schwebel@pengutronix.de> wrote:

>> - Ways to do more automated testing of ARM kernels, including hearing
>>   from other people what they use to build/boot/test/benchmark their
>>   code (perhaps even get a discussion going about using the diverse
>>   range of hardware people have around put to use as test-machines).
>
> That's pretty interesting; we have a test farm at PTX, but it mainly
> does nightly build + boot tests on mainline, for systems we care of.

I have a small (but growing) farm of boards here that I do some (so
far) limited boot testing, but I do it a few times a day on mainline,
and nightly on linux-next and arm-soc for-next branches. I also do it
for -stable but right now only for published releases since I so far
only handle pulling whole git branches.

I also build every defconfig on arm at the same cadence. Everything's
handled by a couple of scripts that emails me the results (as compared
to building a status webpage that I have to go check). So essentially
every morning I have an email with the fresh -next breakage waiting
for me to look at.

The hardware comes from various sources; some I've bought myself,
other I've been sent by kind vendors. All of them are "modern" though,
i.e. v7-class hardware with decent amount of memory, etc. I netboot
everything with local rootfs at the moment.


I honestly don't know if there's much point in doing complex
centralized testing/reporting. I've found that I never go look for the
results. I don't regularly check Russell's autobuilder status, for
example. Nor the kisskb -next build status. But having my own scripts
email me has been useful. I do hope that most ARM subplatform
maintainers have some semi-automated setup for frequent testing as
well, but I know that reality is that far from all do.

It could make sense to show what some of us use to give others
examples of what might work for them. There's a thousand ways to build
these kind of things, some elaborate and others quite simple. I've
definitely started at the simple end myself. :)


-Olof

[ARM ATTEND] Interested in R/M-class (!MMU), automated testing

[Olof Johansson]

[+ksummit-discuss which had fallen off]

On Sun, Aug 4, 2013 at 11:49 PM, Olof Johansson <olof@lixom.net> wrote:
> On Sun, Aug 4, 2013 at 5:25 AM, Robert Schwebel
> <r.schwebel@pengutronix.de> wrote:
>
>>> - Ways to do more automated testing of ARM kernels, including hearing
>>>   from other people what they use to build/boot/test/benchmark their
>>>   code (perhaps even get a discussion going about using the diverse
>>>   range of hardware people have around put to use as test-machines).
>>
>> That's pretty interesting; we have a test farm at PTX, but it mainly
>> does nightly build + boot tests on mainline, for systems we care of.
>
> I have a small (but growing) farm of boards here that I do some (so
> far) limited boot testing, but I do it a few times a day on mainline,
> and nightly on linux-next and arm-soc for-next branches. I also do it
> for -stable but right now only for published releases since I so far
> only handle pulling whole git branches.
>
> I also build every defconfig on arm at the same cadence. Everything's
> handled by a couple of scripts that emails me the results (as compared
> to building a status webpage that I have to go check). So essentially
> every morning I have an email with the fresh -next breakage waiting
> for me to look at.
>
> The hardware comes from various sources; some I've bought myself,
> other I've been sent by kind vendors. All of them are "modern" though,
> i.e. v7-class hardware with decent amount of memory, etc. I netboot
> everything with local rootfs at the moment.
>
>
> I honestly don't know if there's much point in doing complex
> centralized testing/reporting. I've found that I never go look for the
> results. I don't regularly check Russell's autobuilder status, for
> example. Nor the kisskb -next build status. But having my own scripts
> email me has been useful. I do hope that most ARM subplatform
> maintainers have some semi-automated setup for frequent testing as
> well, but I know that reality is that far from all do.
>
> It could make sense to show what some of us use to give others
> examples of what might work for them. There's a thousand ways to build
> these kind of things, some elaborate and others quite simple. I've
> definitely started at the simple end myself. :)
>
>
> -Olof

[ARM ATTEND] Interested in R/M-class (!MMU), automated testing

[Jonathan Austin]

Hi Olof, Robert

(+Paul Walmsley on CC as I mention his recent email below)
On 05/08/13 07:49, Olof Johansson wrote:
> [+ksummit-discuss which had fallen off]
> 
> On Sun, Aug 4, 2013 at 11:49 PM, Olof Johansson <olof@lixom.net> wrote:
>> On Sun, Aug 4, 2013 at 5:25 AM, Robert Schwebel
>> <r.schwebel@pengutronix.de> wrote:
>>
>>>> - Ways to do more automated testing of ARM kernels, including hearing
>>>>    from other people what they use to build/boot/test/benchmark their
>>>>    code (perhaps even get a discussion going about using the diverse
>>>>    range of hardware people have around put to use as test-machines).
>>>
>>> That's pretty interesting; we have a test farm at PTX, but it mainly
>>> does nightly build + boot tests on mainline, for systems we care of.
>>
>> I have a small (but growing) farm of boards here that I do some (so
>> far) limited boot testing, but I do it a few times a day on mainline,
>> and nightly on linux-next and arm-soc for-next branches. I also do it
>> for -stable but right now only for published releases since I so far
>> only handle pulling whole git branches.
>>
>> I also build every defconfig on arm at the same cadence. Everything's
>> handled by a couple of scripts that emails me the results (as compared
>> to building a status webpage that I have to go check). So essentially
>> every morning I have an email with the fresh -next breakage waiting
>> for me to look at.
>>
>> The hardware comes from various sources; some I've bought myself,
>> other I've been sent by kind vendors. All of them are "modern" though,
>> i.e. v7-class hardware with decent amount of memory, etc. I netboot
>> everything with local rootfs at the moment.
>>
>>
>> I honestly don't know if there's much point in doing complex
>> centralized testing/reporting. I've found that I never go look for the
>> results. I don't regularly check Russell's autobuilder status, for

I broadly agree with you here - I find I tend to go and look there
when things are broken ("when did this first break?"), but honestly
wasn't aware of it until very recently and don't poll it.

I think that some level of centralised regression testing on
performance might be nice - if there was notification when things
jumped in a way that bucked the trend, for example, it might be good
to know about. Do your scripts monitor that sort of thing?

The other reason I mention sharing (which I guess implies something
a little centralised) is for the case where the code you are working
on has implications for a core you don't have - I'm not sure there
are many people with 11MPCores around, for example!

(However, I don't think anything that leads people to abdicate their
personal responsibility for testing patches before they go out the door
is a good idea!)

>> example. Nor the kisskb -next build status. But having my own scripts
>> email me has been useful. I do hope that most ARM subplatform
>> maintainers have some semi-automated setup for frequent testing as
>> well, but I know that reality is that far from all do.
>>
>> It could make sense to show what some of us use to give others
>> examples of what might work for them. There's a thousand ways to build
>> these kind of things, some elaborate and others quite simple. I've
>> definitely started at the simple end myself. :)
>>

I think this is more the sort of thing I was hoping to discuss.

I know Paul Walmsley has some pretty functional scripts that he's recently
offered to share part of (assuming he gets time to tidy them up):

http://lists.infradead.org/pipermail/linux-arm-kernel/2013-July/187916.html

So that seems to be at least three separate approaches. Further, I took a
look recently at the ktest.pl script, which looked handy but not so
targetted at the ARM world, where you care about testing the same thing on
a wide array of platforms, might want an NFS root file-system, etc.

Here we tend to have a lot of hardware on our desks, and we test a lot
but mainly on an ad-hoc basis/where we see changes will intersect
with any specific board or core's quirks. Hearing how other people handle
automation would be great :)

Jonny

[Ksummit-2013-discuss] [ARM ATTEND] Interested in R/M-class (!MMU), automated testing

[Steven Rostedt]

On Mon, 2013-08-05 at 11:08 +0100, Jonathan Austin wrote:

> So that seems to be at least three separate approaches. Further, I took a
> look recently at the ktest.pl script, which looked handy but not so
> targetted at the ARM world, where you care about testing the same thing on
> a wide array of platforms, might want an NFS root file-system, etc.

As the maintainer of ktest.pl, I would be very interested in working
with you to make it easier to handle your requirements. This was the
purpose of ktest.pl in the first place :-)

I'll happily attend any of the ARM sessions that talk about testing.

-- Steve

[ARM ATTEND] Interested in R/M-class (!MMU), automated testing

[Aaro Koskinen]

On Sun, Aug 04, 2013 at 11:49:03PM -0700, Olof Johansson wrote:
> On Sun, Aug 4, 2013 at 5:25 AM, Robert Schwebel
> <r.schwebel@pengutronix.de> wrote:
> 
> >> - Ways to do more automated testing of ARM kernels, including hearing
> >>   from other people what they use to build/boot/test/benchmark their
> >>   code (perhaps even get a discussion going about using the diverse
> >>   range of hardware people have around put to use as test-machines).
> >
> > That's pretty interesting; we have a test farm at PTX, but it mainly
> > does nightly build + boot tests on mainline, for systems we care of.
> 
> I have a small (but growing) farm of boards here that I do some (so
> far) limited boot testing, but I do it a few times a day on mainline,
> and nightly on linux-next and arm-soc for-next branches.

How successful is your testing? Any statistics how many pull requests
or patches you can shoot down or revert (or ask to be redone) before
they hit mainline?

A.

[Ksummit-2013-discuss] [ARM ATTEND] Interested in R/M-class (!MMU), automated testing

[Guenter Roeck]

On 08/06/2013 04:44 PM, Aaro Koskinen wrote:
> On Sun, Aug 04, 2013 at 11:49:03PM -0700, Olof Johansson wrote:
>> On Sun, Aug 4, 2013 at 5:25 AM, Robert Schwebel
>> <r.schwebel@pengutronix.de> wrote:
>>
>>>> - Ways to do more automated testing of ARM kernels, including hearing
>>>>    from other people what they use to build/boot/test/benchmark their
>>>>    code (perhaps even get a discussion going about using the diverse
>>>>    range of hardware people have around put to use as test-machines).
>>>
>>> That's pretty interesting; we have a test farm at PTX, but it mainly
>>> does nightly build + boot tests on mainline, for systems we care of.
>>
>> I have a small (but growing) farm of boards here that I do some (so
>> far) limited boot testing, but I do it a few times a day on mainline,
>> and nightly on linux-next and arm-soc for-next branches.
>
> How successful is your testing? Any statistics how many pull requests
> or patches you can shoot down or revert (or ask to be redone) before
> they hit mainline?
>

Someone told me that the Yocto project starts implementing automated tests.
Wonder if those are done for arm targets as well. Either case, if anyone has
experience running basic arm tests on qemu I would be interested to know -
it is on my task list for automated testing on the -stable queue.

Guenter

[ARM ATTEND] Interested in R/M-class (!MMU), automated testing

[Olof Johansson]

On Tue, Aug 6, 2013 at 4:44 PM, Aaro Koskinen <aaro.koskinen@iki.fi> wrote:
> On Sun, Aug 04, 2013 at 11:49:03PM -0700, Olof Johansson wrote:
>> On Sun, Aug 4, 2013 at 5:25 AM, Robert Schwebel
>> <r.schwebel@pengutronix.de> wrote:
>>
>> >> - Ways to do more automated testing of ARM kernels, including hearing
>> >>   from other people what they use to build/boot/test/benchmark their
>> >>   code (perhaps even get a discussion going about using the diverse
>> >>   range of hardware people have around put to use as test-machines).
>> >
>> > That's pretty interesting; we have a test farm at PTX, but it mainly
>> > does nightly build + boot tests on mainline, for systems we care of.
>>
>> I have a small (but growing) farm of boards here that I do some (so
>> far) limited boot testing, but I do it a few times a day on mainline,
>> and nightly on linux-next and arm-soc for-next branches.
>
> How successful is your testing? Any statistics how many pull requests
> or patches you can shoot down or revert (or ask to be redone) before
> they hit mainline?

This is a pretty new thing, I used to do it manually a few times as
week but the fully automated setup has only been online for about a
month. So data is pretty limited so far.

I've been able to catch quite a few things pretty quickly though. A
few boot issues in -next, and Russell's security fixes for -rc4 showed
up quickly so he could do a quick follow-up fix. I've also been
weeding out warnings from the defconfig builds so that new issues are
quicker to spot.

There's a current build breakage on linux-next for bcm2835_defconfig
and mxs_defconfig that needs looking at, btw. :)


-Olof

[ARM ATTEND] Interested in R/M-class (!MMU), automated testing

[Fabio Estevam]

Hi Olof,

On Tue, Aug 6, 2013 at 9:28 PM, Olof Johansson <olof@lixom.net> wrote:

> There's a current build breakage on linux-next for bcm2835_defconfig
> and mxs_defconfig that needs looking at, btw. :)

This issue is fixed by this patch:
http://ozlabs.org/~akpm/mmots/broken-out/watchdog-update-watchdog_tresh-properly-fix.patch

[Ksummit-2013-discuss] [ARM ATTEND] Interested in R/M-class (!MMU), automated testing

[Jonathan Austin]

On 05/08/13 18:58, Steven Rostedt wrote:
> On Mon, 2013-08-05 at 11:08 +0100, Jonathan Austin wrote:
>
>> So that seems to be at least three separate approaches. Further, I took a
>> look recently at the ktest.pl script, which looked handy but not so
>> targetted at the ARM world, where you care about testing the same thing on
>> a wide array of platforms, might want an NFS root file-system, etc.
>
> As the maintainer of ktest.pl, I would be very interested in working
> with you to make it easier to handle your requirements. This was the
> purpose of ktest.pl in the first place :-)

Thanks very much - I don't know if I'm the only one that's looked at 
ktest, but hopefully some of the others that already have fuller systems 
also have thoughts on this. I'll be sure to have a more thorough look at 
how this might work beforehand, so that if I am at the summit I can have 
some more input.

>
> I'll happily attend any of the ARM sessions that talk about testing.
>

Olof, perhaps your idea of sharing what you've recently set up could 
form the core of a session? From your other email it looks like that 
system is already being quite useful...

Jonny