Re: next boot: 564 boots: 59 failed, 448 passed with 57 offline (next-20160107)

Re: next boot: 564 boots: 59 failed, 448 passed with 57 offline (next-20160107)

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 2417 bytes --]

On Thu, Jan 07, 2016 at 07:41:52AM -0800, kernelci.org bot wrote:

> Tree: next
> Branch: local/master
> Git Describe: next-20160107
> Git Commit: 0f023a298a7d2a8e99c54913f7277b3c08c1f5c9
> Git URL: git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
> Tested: 95 unique boards, 25 SoC families, 28 builds out of 135
> 
> Boot Failures Detected: https://kernelci.org/boot/?next-20160107&fail

For the past couple of days -next has been failing to boot on various
Tegra platforms, there are failures involving either no kernel output or
(with tegra_defconfig) backtraces during CMA allocation from USB, for
example:

   http://storage.kernelci.org/next/next-20160107/arm-tegra_defconfig/lab-khilman/boot-tegra30-beaver.html

>     multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y:
>         tegra124-jetson-tk1: 2 failed labs
>         tegra30-beaver: 1 failed lab
> 
>     tegra_defconfig:
>         tegra124-jetson-tk1: 2 failed labs
>         tegra124-jetson-tk1_rootfs:nfs: 1 failed lab
>         tegra124-nyan-big: 1 failed lab
>         tegra30-beaver: 1 failed lab
> 
>     multi_v7_defconfig+CONFIG_ARM_LPAE=y:
>         tegra124-jetson-tk1: 1 failed lab
>         tegra124-nyan-big: 1 failed lab
> 
>     multi_v7_defconfig:
>         tegra124-jetson-tk1: 2 failed labs
>         tegra30-beaver: 1 failed lab
> 
>     multi_v7_defconfig+CONFIG_LKDTM=y:
>         tegra124-jetson-tk1: 2 failed labs
>         tegra30-beaver: 1 failed lab
> 
>     multi_v7_defconfig+CONFIG_PROVE_LOCKING=y:
>         tegra124-jetson-tk1: 2 failed labs
>         tegra30-beaver: 1 failed lab

I did bisects yesterday but they were a bit inconclusive, showing a
series of skipped commits in the TTY tree as being the culprits:

afd7f88f157796e586fc99d62da13a54024e0731 serial: 8250: move of_serial code to 8250 directory
4e33870b3bb691996354a8f9e8f69458b4fc34d9 serial: of: CONFIG_SERIAL_8250 is always set
d1b5c87fa8058a3f477ae05555916dd1cea934ad serial: remove NWP serial support

which seems more than a little surprising and like they might be false
positives, especially in the cases that do produce output.  The errors
today look the same.  I've not investigated further than the bisects.

Full bisect logs:

 https://ci.linaro.org/view/people/job/tbaker-boot-bisect-bot/140/console
 https://ci.linaro.org/view/people/job/tbaker-boot-bisect-bot/141/console

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: next boot: 564 boots: 59 failed, 448 passed with 57 offline (next-20160107)

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 3140 bytes --]

On Thu, Jan 07, 2016 at 03:59:58PM +0000, Mark Brown wrote:
> On Thu, Jan 07, 2016 at 07:41:52AM -0800, kernelci.org bot wrote:
> 
> > Tree: next
> > Branch: local/master
> > Git Describe: next-20160107
> > Git Commit: 0f023a298a7d2a8e99c54913f7277b3c08c1f5c9
> > Git URL: git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
> > Tested: 95 unique boards, 25 SoC families, 28 builds out of 135
> > 
> > Boot Failures Detected: https://kernelci.org/boot/?next-20160107&fail
> 
> For the past couple of days -next has been failing to boot on various
> Tegra platforms, there are failures involving either no kernel output or
> (with tegra_defconfig) backtraces during CMA allocation from USB, for
> example:
> 
>    http://storage.kernelci.org/next/next-20160107/arm-tegra_defconfig/lab-khilman/boot-tegra30-beaver.html
> 
> >     multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y:
> >         tegra124-jetson-tk1: 2 failed labs
> >         tegra30-beaver: 1 failed lab
> > 
> >     tegra_defconfig:
> >         tegra124-jetson-tk1: 2 failed labs
> >         tegra124-jetson-tk1_rootfs:nfs: 1 failed lab
> >         tegra124-nyan-big: 1 failed lab
> >         tegra30-beaver: 1 failed lab
> > 
> >     multi_v7_defconfig+CONFIG_ARM_LPAE=y:
> >         tegra124-jetson-tk1: 1 failed lab
> >         tegra124-nyan-big: 1 failed lab
> > 
> >     multi_v7_defconfig:
> >         tegra124-jetson-tk1: 2 failed labs
> >         tegra30-beaver: 1 failed lab
> > 
> >     multi_v7_defconfig+CONFIG_LKDTM=y:
> >         tegra124-jetson-tk1: 2 failed labs
> >         tegra30-beaver: 1 failed lab
> > 
> >     multi_v7_defconfig+CONFIG_PROVE_LOCKING=y:
> >         tegra124-jetson-tk1: 2 failed labs
> >         tegra30-beaver: 1 failed lab
> 
> I did bisects yesterday but they were a bit inconclusive, showing a
> series of skipped commits in the TTY tree as being the culprits:
> 
> afd7f88f157796e586fc99d62da13a54024e0731 serial: 8250: move of_serial code to 8250 directory
> 4e33870b3bb691996354a8f9e8f69458b4fc34d9 serial: of: CONFIG_SERIAL_8250 is always set
> d1b5c87fa8058a3f477ae05555916dd1cea934ad serial: remove NWP serial support
> 
> which seems more than a little surprising and like they might be false
> positives, especially in the cases that do produce output.  The errors
> today look the same.  I've not investigated further than the bisects.
> 
> Full bisect logs:
> 
>  https://ci.linaro.org/view/people/job/tbaker-boot-bisect-bot/140/console
>  https://ci.linaro.org/view/people/job/tbaker-boot-bisect-bot/141/console

These should all be fixed by this patch:

	serial: 8250: of: Fix the driver and actually compile the 8250_of

which can be found in patchwork here:

	https://patchwork.kernel.org/patch/7883541/

I think this is merely waiting to be picked up into the tty tree. I
suspect the reason why there's output in the first place is because
of earlyprintk. I've seen recent linux-next work properly, with the
serial console hanging at some point (fbcon shows login prompt). If
I apply the above patch things are back to normal.

Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: next boot: 564 boots: 59 failed, 448 passed with 57 offline (next-20160107)

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 504 bytes --]

On Thu, Jan 07, 2016 at 05:28:10PM +0100, Thierry Reding wrote:

> These should all be fixed by this patch:

> 	serial: 8250: of: Fix the driver and actually compile the 8250_of

> which can be found in patchwork here:

> 	https://patchwork.kernel.org/patch/7883541/

> I think this is merely waiting to be picked up into the tty tree. I

Ah, good - hopefully that'll get picked up quickly, I suspect it's going
to turn out to be responsible for a bunch of the other failures we're
seeing at the minute.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: next boot: 564 boots: 59 failed, 448 passed with 57 offline (next-20160107)

[Greg Kroah-Hartman]

On Thu, Jan 07, 2016 at 05:28:10PM +0100, Thierry Reding wrote:
> On Thu, Jan 07, 2016 at 03:59:58PM +0000, Mark Brown wrote:
> > On Thu, Jan 07, 2016 at 07:41:52AM -0800, kernelci.org bot wrote:
> > 
> > > Tree: next
> > > Branch: local/master
> > > Git Describe: next-20160107
> > > Git Commit: 0f023a298a7d2a8e99c54913f7277b3c08c1f5c9
> > > Git URL: git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
> > > Tested: 95 unique boards, 25 SoC families, 28 builds out of 135
> > > 
> > > Boot Failures Detected: https://kernelci.org/boot/?next-20160107&fail
> > 
> > For the past couple of days -next has been failing to boot on various
> > Tegra platforms, there are failures involving either no kernel output or
> > (with tegra_defconfig) backtraces during CMA allocation from USB, for
> > example:
> > 
> >    http://storage.kernelci.org/next/next-20160107/arm-tegra_defconfig/lab-khilman/boot-tegra30-beaver.html
> > 
> > >     multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y:
> > >         tegra124-jetson-tk1: 2 failed labs
> > >         tegra30-beaver: 1 failed lab
> > > 
> > >     tegra_defconfig:
> > >         tegra124-jetson-tk1: 2 failed labs
> > >         tegra124-jetson-tk1_rootfs:nfs: 1 failed lab
> > >         tegra124-nyan-big: 1 failed lab
> > >         tegra30-beaver: 1 failed lab
> > > 
> > >     multi_v7_defconfig+CONFIG_ARM_LPAE=y:
> > >         tegra124-jetson-tk1: 1 failed lab
> > >         tegra124-nyan-big: 1 failed lab
> > > 
> > >     multi_v7_defconfig:
> > >         tegra124-jetson-tk1: 2 failed labs
> > >         tegra30-beaver: 1 failed lab
> > > 
> > >     multi_v7_defconfig+CONFIG_LKDTM=y:
> > >         tegra124-jetson-tk1: 2 failed labs
> > >         tegra30-beaver: 1 failed lab
> > > 
> > >     multi_v7_defconfig+CONFIG_PROVE_LOCKING=y:
> > >         tegra124-jetson-tk1: 2 failed labs
> > >         tegra30-beaver: 1 failed lab
> > 
> > I did bisects yesterday but they were a bit inconclusive, showing a
> > series of skipped commits in the TTY tree as being the culprits:
> > 
> > afd7f88f157796e586fc99d62da13a54024e0731 serial: 8250: move of_serial code to 8250 directory
> > 4e33870b3bb691996354a8f9e8f69458b4fc34d9 serial: of: CONFIG_SERIAL_8250 is always set
> > d1b5c87fa8058a3f477ae05555916dd1cea934ad serial: remove NWP serial support
> > 
> > which seems more than a little surprising and like they might be false
> > positives, especially in the cases that do produce output.  The errors
> > today look the same.  I've not investigated further than the bisects.
> > 
> > Full bisect logs:
> > 
> >  https://ci.linaro.org/view/people/job/tbaker-boot-bisect-bot/140/console
> >  https://ci.linaro.org/view/people/job/tbaker-boot-bisect-bot/141/console
> 
> These should all be fixed by this patch:
> 
> 	serial: 8250: of: Fix the driver and actually compile the 8250_of
> 
> which can be found in patchwork here:
> 
> 	https://patchwork.kernel.org/patch/7883541/
> 
> I think this is merely waiting to be picked up into the tty tree. I
> suspect the reason why there's output in the first place is because
> of earlyprintk. I've seen recent linux-next work properly, with the
> serial console hanging at some point (fbcon shows login prompt). If
> I apply the above patch things are back to normal.

I'll go queue that up later, but how is this suddenly showing up in
-next?  The tty tree has been pretty quiet for a while now..

thanks,

greg k-h

Re: next boot: 564 boots: 59 failed, 448 passed with 57 offline (next-20160107)

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 906 bytes --]

On Thu, Jan 07, 2016 at 10:57:39AM -0800, Greg Kroah-Hartman wrote:
> On Thu, Jan 07, 2016 at 05:28:10PM +0100, Thierry Reding wrote:
> > On Thu, Jan 07, 2016 at 03:59:58PM +0000, Mark Brown wrote:

> > I think this is merely waiting to be picked up into the tty tree. I
> > suspect the reason why there's output in the first place is because
> > of earlyprintk. I've seen recent linux-next work properly, with the
> > serial console hanging at some point (fbcon shows login prompt). If
> > I apply the above patch things are back to normal.

> I'll go queue that up later, but how is this suddenly showing up in
> -next?  The tty tree has been pretty quiet for a while now..

There were some memory management issues which broke booting on almost
all ARM platforms for most of this week so nothing else was really
visible and relatively few -next builds over the Christmas vacation (as
one would expect).

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]