Re: linux-next: Tree for Mar 14 (mips qemu failure bisected)

Re: linux-next: Tree for Mar 14 (mips qemu failure bisected)

[Guenter Roeck]

On Mon, Mar 14, 2016 at 07:37:29AM -0700, Guenter Roeck wrote:
> On Mon, Mar 14, 2016 at 05:40:37PM +1100, Stephen Rothwell wrote:
> > Hi all,
> > 
> > Changes since 20160311:
> > 
> > The vfs tree gained a conflict against Linus' tree. I also applied a
> > patch for a known runtime bug.
> > 
> > The tip tree gained a conflict against the mips tree.
> > 
> > The aio tree still had a build failure so I removed several commits
> > from it.  It also gained a conflict against the vfs tree.
> > 
> > Non-merge commits (relative to Linus' tree): 11202
> >  8646 files changed, 426680 insertions(+), 211740 deletions(-)
> > 
> 
> To give people an idea what to expect in the merge window, here are my current
> build and runtime test results. Some of the runtime failures are due to the
> newly introduced i2c bug, but many (including the arm64 boot failures) have
> been around for a while.
> 
[ ... ]

> Qemu test results:
> 	total: 96 pass: 69 fail: 27
> Failed tests:
[ ... ]
> 	mips:mips_malta_smp_defconfig

I bisected this failure to commit bb11cff327e54 ("MIPS: Make smp CMP, CPS and MT
use the new generic IPI functions". Bisect log is attached.

> 	mips64:smp:mips_malta64_defconfig
> 	mips:mipsel_malta_smp_defconfig
> 	mips:mipsel_malta64_smp_defconfig

If necessary I can repeat the bisect for those. Please let me know.

Thanks,
Guenter

---
Bisect log:

# bad: [4342eec3c5a2402ca5de3d6e56f541fe1c5171e2] Add linux-next specific files for 20160314
# good: [f6cede5b49e822ebc41a099fe41ab4989f64e2cb] Linux 4.5-rc7
git bisect start 'HEAD' 'v4.5-rc7'
# good: [0525c3e26ec2c43cd509433be3be25210a0154ef] Merge remote-tracking branch 'drm-tegra/drm/tegra/for-next'
git bisect good 0525c3e26ec2c43cd509433be3be25210a0154ef
# bad: [385128a1b49762c1b9515c9f6294aeebbc55b956] Merge remote-tracking branch 'usb-chipidea-next/ci-for-usb-next'
git bisect bad 385128a1b49762c1b9515c9f6294aeebbc55b956
# good: [dfdb27baab4fc45c9399a991270413d0fb1c694a] Merge remote-tracking branch 'spi/for-next'
git bisect good dfdb27baab4fc45c9399a991270413d0fb1c694a
# bad: [e368d7d2a0dce6d6795ead1fc8a09bcca8a4a565] Merge branch 'timers/nohz'
git bisect bad e368d7d2a0dce6d6795ead1fc8a09bcca8a4a565
# good: [ced30bc9129777d715057d06fc8dbdfd3b81e94d] Merge tag 'perf-core-for-mingo-20160310' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux into perf/core
git bisect good ced30bc9129777d715057d06fc8dbdfd3b81e94d
# bad: [656a61d4d9cbb8dfc2d007281190b2eccebad522] manual merge of mm/pkeys
git bisect bad 656a61d4d9cbb8dfc2d007281190b2eccebad522
# good: [16f7379f2da43f29d9faa2f474745e2705a3f510] Merge branch 'efi/core'
git bisect good 16f7379f2da43f29d9faa2f474745e2705a3f510
# bad: [a7fb9a8169be9a55e0cfb98346aece1b51c016fa] Merge branch 'locking/core'
git bisect bad a7fb9a8169be9a55e0cfb98346aece1b51c016fa
# good: [2a07870511829977d02609dac6450017b0419ea9] irqchip/mips-gic: Use gic_vpes instead of NR_CPUS
git bisect good 2a07870511829977d02609dac6450017b0419ea9
# good: [eaff0e7003cca6c2748b67ead2d4b1a8ad858fc7] locking/pvqspinlock: Move lock stealing count tracking code into pv_queued_spin_steal_lock()
git bisect good eaff0e7003cca6c2748b67ead2d4b1a8ad858fc7
# good: [013e379a3094ff2898f8d33cfbff1573d471ee14] tools/lib/lockdep: Fix link creation warning
git bisect good 013e379a3094ff2898f8d33cfbff1573d471ee14
# bad: [7eb8c99db26cc6499bfb1eba72dffc4730570752] MIPS: Delete smp-gic.c
git bisect bad 7eb8c99db26cc6499bfb1eba72dffc4730570752
# good: [fbde2d7d8290d8c642d591a471356abda2446874] MIPS: Add generic SMP IPI support
git bisect good fbde2d7d8290d8c642d591a471356abda2446874
# bad: [bb11cff327e54179c13446c4022ed4ed7d4871c7] MIPS: Make smp CMP, CPS and MT use the new generic IPI functions
git bisect bad bb11cff327e54179c13446c4022ed4ed7d4871c7
# first bad commit: [bb11cff327e54179c13446c4022ed4ed7d4871c7] MIPS: Make smp CMP, CPS and MT use the new generic IPI functions

Re: linux-next: Tree for Mar 14 (mips qemu failure bisected)

[Qais Yousef]

Hi Guenter,

On 15/03/2016 05:26, Guenter Roeck wrote:
> On Mon, Mar 14, 2016 at 07:37:29AM -0700, Guenter Roeck wrote:
>> On Mon, Mar 14, 2016 at 05:40:37PM +1100, Stephen Rothwell wrote:
>>> Hi all,
>>>
>>> Changes since 20160311:
>>>
>>> The vfs tree gained a conflict against Linus' tree. I also applied a
>>> patch for a known runtime bug.
>>>
>>> The tip tree gained a conflict against the mips tree.
>>>
>>> The aio tree still had a build failure so I removed several commits
>>> from it.  It also gained a conflict against the vfs tree.
>>>
>>> Non-merge commits (relative to Linus' tree): 11202
>>>   8646 files changed, 426680 insertions(+), 211740 deletions(-)
>>>
>> To give people an idea what to expect in the merge window, here are my current
>> build and runtime test results. Some of the runtime failures are due to the
>> newly introduced i2c bug, but many (including the arm64 boot failures) have
>> been around for a while.
>>
> [ ... ]
>
>> Qemu test results:
>> 	total: 96 pass: 69 fail: 27
>> Failed tests:
> [ ... ]
>> 	mips:mips_malta_smp_defconfig
> I bisected this failure to commit bb11cff327e54 ("MIPS: Make smp CMP, CPS and MT
> use the new generic IPI functions". Bisect log is attached.

Thanks for bisecting this. I tested this on a real Malta system but not 
qemu. I'll try to reproduce.

Can I get qemu run script and the instructions to use it from somewhere?

Thanks,
Qais

>
>> 	mips64:smp:mips_malta64_defconfig
>> 	mips:mipsel_malta_smp_defconfig
>> 	mips:mipsel_malta64_smp_defconfig
> If necessary I can repeat the bisect for those. Please let me know.
>
> Thanks,
> Guenter
>
> ---
> Bisect log:
>
> # bad: [4342eec3c5a2402ca5de3d6e56f541fe1c5171e2] Add linux-next specific files for 20160314
> # good: [f6cede5b49e822ebc41a099fe41ab4989f64e2cb] Linux 4.5-rc7
> git bisect start 'HEAD' 'v4.5-rc7'
> # good: [0525c3e26ec2c43cd509433be3be25210a0154ef] Merge remote-tracking branch 'drm-tegra/drm/tegra/for-next'
> git bisect good 0525c3e26ec2c43cd509433be3be25210a0154ef
> # bad: [385128a1b49762c1b9515c9f6294aeebbc55b956] Merge remote-tracking branch 'usb-chipidea-next/ci-for-usb-next'
> git bisect bad 385128a1b49762c1b9515c9f6294aeebbc55b956
> # good: [dfdb27baab4fc45c9399a991270413d0fb1c694a] Merge remote-tracking branch 'spi/for-next'
> git bisect good dfdb27baab4fc45c9399a991270413d0fb1c694a
> # bad: [e368d7d2a0dce6d6795ead1fc8a09bcca8a4a565] Merge branch 'timers/nohz'
> git bisect bad e368d7d2a0dce6d6795ead1fc8a09bcca8a4a565
> # good: [ced30bc9129777d715057d06fc8dbdfd3b81e94d] Merge tag 'perf-core-for-mingo-20160310' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux into perf/core
> git bisect good ced30bc9129777d715057d06fc8dbdfd3b81e94d
> # bad: [656a61d4d9cbb8dfc2d007281190b2eccebad522] manual merge of mm/pkeys
> git bisect bad 656a61d4d9cbb8dfc2d007281190b2eccebad522
> # good: [16f7379f2da43f29d9faa2f474745e2705a3f510] Merge branch 'efi/core'
> git bisect good 16f7379f2da43f29d9faa2f474745e2705a3f510
> # bad: [a7fb9a8169be9a55e0cfb98346aece1b51c016fa] Merge branch 'locking/core'
> git bisect bad a7fb9a8169be9a55e0cfb98346aece1b51c016fa
> # good: [2a07870511829977d02609dac6450017b0419ea9] irqchip/mips-gic: Use gic_vpes instead of NR_CPUS
> git bisect good 2a07870511829977d02609dac6450017b0419ea9
> # good: [eaff0e7003cca6c2748b67ead2d4b1a8ad858fc7] locking/pvqspinlock: Move lock stealing count tracking code into pv_queued_spin_steal_lock()
> git bisect good eaff0e7003cca6c2748b67ead2d4b1a8ad858fc7
> # good: [013e379a3094ff2898f8d33cfbff1573d471ee14] tools/lib/lockdep: Fix link creation warning
> git bisect good 013e379a3094ff2898f8d33cfbff1573d471ee14
> # bad: [7eb8c99db26cc6499bfb1eba72dffc4730570752] MIPS: Delete smp-gic.c
> git bisect bad 7eb8c99db26cc6499bfb1eba72dffc4730570752
> # good: [fbde2d7d8290d8c642d591a471356abda2446874] MIPS: Add generic SMP IPI support
> git bisect good fbde2d7d8290d8c642d591a471356abda2446874
> # bad: [bb11cff327e54179c13446c4022ed4ed7d4871c7] MIPS: Make smp CMP, CPS and MT use the new generic IPI functions
> git bisect bad bb11cff327e54179c13446c4022ed4ed7d4871c7
> # first bad commit: [bb11cff327e54179c13446c4022ed4ed7d4871c7] MIPS: Make smp CMP, CPS and MT use the new generic IPI functions

Re: linux-next: Tree for Mar 14 (mips qemu failure bisected)

[Guenter Roeck]

On Tue, Mar 15, 2016 at 09:55:06PM +0000, Qais Yousef wrote:
> Hi Guenter,
> 
> On 15/03/2016 05:26, Guenter Roeck wrote:
> >On Mon, Mar 14, 2016 at 07:37:29AM -0700, Guenter Roeck wrote:
> >>On Mon, Mar 14, 2016 at 05:40:37PM +1100, Stephen Rothwell wrote:
> >>>Hi all,
> >>>
> >>>Changes since 20160311:
> >>>
> >>>The vfs tree gained a conflict against Linus' tree. I also applied a
> >>>patch for a known runtime bug.
> >>>
> >>>The tip tree gained a conflict against the mips tree.
> >>>
> >>>The aio tree still had a build failure so I removed several commits
> >>>from it.  It also gained a conflict against the vfs tree.
> >>>
> >>>Non-merge commits (relative to Linus' tree): 11202
> >>>  8646 files changed, 426680 insertions(+), 211740 deletions(-)
> >>>
> >>To give people an idea what to expect in the merge window, here are my current
> >>build and runtime test results. Some of the runtime failures are due to the
> >>newly introduced i2c bug, but many (including the arm64 boot failures) have
> >>been around for a while.
> >>
> >[ ... ]
> >
> >>Qemu test results:
> >>	total: 96 pass: 69 fail: 27
> >>Failed tests:
> >[ ... ]
> >>	mips:mips_malta_smp_defconfig
> >I bisected this failure to commit bb11cff327e54 ("MIPS: Make smp CMP, CPS and MT
> >use the new generic IPI functions". Bisect log is attached.
> 
> Thanks for bisecting this. I tested this on a real Malta system but not
> qemu. I'll try to reproduce.
> 
I run the tests with only a single CPU core enabled. Maybe that causes
problems with your code ?

> Can I get qemu run script and the instructions to use it from somewhere?
> 
Clone https://github.com/groeck/linux-build-test.git.
Look in rootfs/*mips* for qemu scripts and root file systems.
The SMP tests all fail, so it should not matter which one you pick.
I use my own version of qemu, but for mips standard qemu 2.5.0 should
work fine (I don't have any mips specific changes in my version).

Hope this helps,
Guenter

> Thanks,
> Qais
> 
> >
> >>	mips64:smp:mips_malta64_defconfig
> >>	mips:mipsel_malta_smp_defconfig
> >>	mips:mipsel_malta64_smp_defconfig
> >If necessary I can repeat the bisect for those. Please let me know.
> >
> >Thanks,
> >Guenter
> >
> >---
> >Bisect log:
> >
> ># bad: [4342eec3c5a2402ca5de3d6e56f541fe1c5171e2] Add linux-next specific files for 20160314
> ># good: [f6cede5b49e822ebc41a099fe41ab4989f64e2cb] Linux 4.5-rc7
> >git bisect start 'HEAD' 'v4.5-rc7'
> ># good: [0525c3e26ec2c43cd509433be3be25210a0154ef] Merge remote-tracking branch 'drm-tegra/drm/tegra/for-next'
> >git bisect good 0525c3e26ec2c43cd509433be3be25210a0154ef
> ># bad: [385128a1b49762c1b9515c9f6294aeebbc55b956] Merge remote-tracking branch 'usb-chipidea-next/ci-for-usb-next'
> >git bisect bad 385128a1b49762c1b9515c9f6294aeebbc55b956
> ># good: [dfdb27baab4fc45c9399a991270413d0fb1c694a] Merge remote-tracking branch 'spi/for-next'
> >git bisect good dfdb27baab4fc45c9399a991270413d0fb1c694a
> ># bad: [e368d7d2a0dce6d6795ead1fc8a09bcca8a4a565] Merge branch 'timers/nohz'
> >git bisect bad e368d7d2a0dce6d6795ead1fc8a09bcca8a4a565
> ># good: [ced30bc9129777d715057d06fc8dbdfd3b81e94d] Merge tag 'perf-core-for-mingo-20160310' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux into perf/core
> >git bisect good ced30bc9129777d715057d06fc8dbdfd3b81e94d
> ># bad: [656a61d4d9cbb8dfc2d007281190b2eccebad522] manual merge of mm/pkeys
> >git bisect bad 656a61d4d9cbb8dfc2d007281190b2eccebad522
> ># good: [16f7379f2da43f29d9faa2f474745e2705a3f510] Merge branch 'efi/core'
> >git bisect good 16f7379f2da43f29d9faa2f474745e2705a3f510
> ># bad: [a7fb9a8169be9a55e0cfb98346aece1b51c016fa] Merge branch 'locking/core'
> >git bisect bad a7fb9a8169be9a55e0cfb98346aece1b51c016fa
> ># good: [2a07870511829977d02609dac6450017b0419ea9] irqchip/mips-gic: Use gic_vpes instead of NR_CPUS
> >git bisect good 2a07870511829977d02609dac6450017b0419ea9
> ># good: [eaff0e7003cca6c2748b67ead2d4b1a8ad858fc7] locking/pvqspinlock: Move lock stealing count tracking code into pv_queued_spin_steal_lock()
> >git bisect good eaff0e7003cca6c2748b67ead2d4b1a8ad858fc7
> ># good: [013e379a3094ff2898f8d33cfbff1573d471ee14] tools/lib/lockdep: Fix link creation warning
> >git bisect good 013e379a3094ff2898f8d33cfbff1573d471ee14
> ># bad: [7eb8c99db26cc6499bfb1eba72dffc4730570752] MIPS: Delete smp-gic.c
> >git bisect bad 7eb8c99db26cc6499bfb1eba72dffc4730570752
> ># good: [fbde2d7d8290d8c642d591a471356abda2446874] MIPS: Add generic SMP IPI support
> >git bisect good fbde2d7d8290d8c642d591a471356abda2446874
> ># bad: [bb11cff327e54179c13446c4022ed4ed7d4871c7] MIPS: Make smp CMP, CPS and MT use the new generic IPI functions
> >git bisect bad bb11cff327e54179c13446c4022ed4ed7d4871c7
> ># first bad commit: [bb11cff327e54179c13446c4022ed4ed7d4871c7] MIPS: Make smp CMP, CPS and MT use the new generic IPI functions
> 

Re: linux-next: Tree for Mar 14 (mips qemu failure bisected)

[Guenter Roeck]

On Tue, Mar 15, 2016 at 05:17:13PM -0700, Guenter Roeck wrote:
> On Tue, Mar 15, 2016 at 09:55:06PM +0000, Qais Yousef wrote:
> > Hi Guenter,
> > 
[ ... ]
> > >
> > >>Qemu test results:
> > >>	total: 96 pass: 69 fail: 27
> > >>Failed tests:
> > >[ ... ]
> > >>	mips:mips_malta_smp_defconfig
> > >I bisected this failure to commit bb11cff327e54 ("MIPS: Make smp CMP, CPS and MT
> > >use the new generic IPI functions". Bisect log is attached.
> > 
> > Thanks for bisecting this. I tested this on a real Malta system but not
> > qemu. I'll try to reproduce.
> > 
> I run the tests with only a single CPU core enabled. Maybe that causes
> problems with your code ?
> 
I ran another qemu test (this time on mainline) with "-smp 2", but the only
difference is that the image now gets stuck even earlier.

Also, I ran another set of bisects, this time with both mips and mips64
on mainline (after your patch landed), with the same results.

Guenter

> > Can I get qemu run script and the instructions to use it from somewhere?
> > 
> Clone https://github.com/groeck/linux-build-test.git.
> Look in rootfs/*mips* for qemu scripts and root file systems.
> The SMP tests all fail, so it should not matter which one you pick.
> I use my own version of qemu, but for mips standard qemu 2.5.0 should
> work fine (I don't have any mips specific changes in my version).
> 
> Hope this helps,
> Guenter
> 
> > Thanks,
> > Qais
> > 
> > >
> > >>	mips64:smp:mips_malta64_defconfig
> > >>	mips:mipsel_malta_smp_defconfig
> > >>	mips:mipsel_malta64_smp_defconfig
> > >If necessary I can repeat the bisect for those. Please let me know.
> > >
> > >Thanks,
> > >Guenter
> > >
> > >---
> > >Bisect log:
> > >
> > ># bad: [4342eec3c5a2402ca5de3d6e56f541fe1c5171e2] Add linux-next specific files for 20160314
> > ># good: [f6cede5b49e822ebc41a099fe41ab4989f64e2cb] Linux 4.5-rc7
> > >git bisect start 'HEAD' 'v4.5-rc7'
> > ># good: [0525c3e26ec2c43cd509433be3be25210a0154ef] Merge remote-tracking branch 'drm-tegra/drm/tegra/for-next'
> > >git bisect good 0525c3e26ec2c43cd509433be3be25210a0154ef
> > ># bad: [385128a1b49762c1b9515c9f6294aeebbc55b956] Merge remote-tracking branch 'usb-chipidea-next/ci-for-usb-next'
> > >git bisect bad 385128a1b49762c1b9515c9f6294aeebbc55b956
> > ># good: [dfdb27baab4fc45c9399a991270413d0fb1c694a] Merge remote-tracking branch 'spi/for-next'
> > >git bisect good dfdb27baab4fc45c9399a991270413d0fb1c694a
> > ># bad: [e368d7d2a0dce6d6795ead1fc8a09bcca8a4a565] Merge branch 'timers/nohz'
> > >git bisect bad e368d7d2a0dce6d6795ead1fc8a09bcca8a4a565
> > ># good: [ced30bc9129777d715057d06fc8dbdfd3b81e94d] Merge tag 'perf-core-for-mingo-20160310' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux into perf/core
> > >git bisect good ced30bc9129777d715057d06fc8dbdfd3b81e94d
> > ># bad: [656a61d4d9cbb8dfc2d007281190b2eccebad522] manual merge of mm/pkeys
> > >git bisect bad 656a61d4d9cbb8dfc2d007281190b2eccebad522
> > ># good: [16f7379f2da43f29d9faa2f474745e2705a3f510] Merge branch 'efi/core'
> > >git bisect good 16f7379f2da43f29d9faa2f474745e2705a3f510
> > ># bad: [a7fb9a8169be9a55e0cfb98346aece1b51c016fa] Merge branch 'locking/core'
> > >git bisect bad a7fb9a8169be9a55e0cfb98346aece1b51c016fa
> > ># good: [2a07870511829977d02609dac6450017b0419ea9] irqchip/mips-gic: Use gic_vpes instead of NR_CPUS
> > >git bisect good 2a07870511829977d02609dac6450017b0419ea9
> > ># good: [eaff0e7003cca6c2748b67ead2d4b1a8ad858fc7] locking/pvqspinlock: Move lock stealing count tracking code into pv_queued_spin_steal_lock()
> > >git bisect good eaff0e7003cca6c2748b67ead2d4b1a8ad858fc7
> > ># good: [013e379a3094ff2898f8d33cfbff1573d471ee14] tools/lib/lockdep: Fix link creation warning
> > >git bisect good 013e379a3094ff2898f8d33cfbff1573d471ee14
> > ># bad: [7eb8c99db26cc6499bfb1eba72dffc4730570752] MIPS: Delete smp-gic.c
> > >git bisect bad 7eb8c99db26cc6499bfb1eba72dffc4730570752
> > ># good: [fbde2d7d8290d8c642d591a471356abda2446874] MIPS: Add generic SMP IPI support
> > >git bisect good fbde2d7d8290d8c642d591a471356abda2446874
> > ># bad: [bb11cff327e54179c13446c4022ed4ed7d4871c7] MIPS: Make smp CMP, CPS and MT use the new generic IPI functions
> > >git bisect bad bb11cff327e54179c13446c4022ed4ed7d4871c7
> > ># first bad commit: [bb11cff327e54179c13446c4022ed4ed7d4871c7] MIPS: Make smp CMP, CPS and MT use the new generic IPI functions
> > 

Re: linux-next: Tree for Mar 14 (mips qemu failure bisected)

[Qais Yousef]

On 16/03/2016 13:22, Guenter Roeck wrote:
> On Tue, Mar 15, 2016 at 05:17:13PM -0700, Guenter Roeck wrote:
>> On Tue, Mar 15, 2016 at 09:55:06PM +0000, Qais Yousef wrote:
>>> Hi Guenter,
>>>
> [ ... ]
>>>>> Qemu test results:
>>>>> 	total: 96 pass: 69 fail: 27
>>>>> Failed tests:
>>>> [ ... ]
>>>>> 	mips:mips_malta_smp_defconfig
>>>> I bisected this failure to commit bb11cff327e54 ("MIPS: Make smp CMP, CPS and MT
>>>> use the new generic IPI functions". Bisect log is attached.
>>> Thanks for bisecting this. I tested this on a real Malta system but not
>>> qemu. I'll try to reproduce.
>>>
>> I run the tests with only a single CPU core enabled. Maybe that causes
>> problems with your code ?
>>
> I ran another qemu test (this time on mainline) with "-smp 2", but the only
> difference is that the image now gets stuck even earlier.
>
> Also, I ran another set of bisects, this time with both mips and mips64
> on mainline (after your patch landed), with the same results.
>
> Guenter
>

OK thanks for the info. The offending commit just enables using quite a 
few of the newly added code before that. So the problem could be in any 
of the newly added code.

Unfortunately I can only look at this during my limited time in the 
evening and I have to setup my system to compile and run this, so I 
won't be able to get to the bottom of this as fast as I'd like to.

Qais

Re: linux-next: Tree for Mar 14 (mips qemu failure bisected)

[Qais Yousef]

[-- Attachment #1: Type: text/plain, Size: 2134 bytes --]

On 16/03/2016 20:27, Qais Yousef wrote:
>
>
> On 16/03/2016 13:22, Guenter Roeck wrote:
>> On Tue, Mar 15, 2016 at 05:17:13PM -0700, Guenter Roeck wrote:
>>> On Tue, Mar 15, 2016 at 09:55:06PM +0000, Qais Yousef wrote:
>>>> Hi Guenter,
>>>>
>> [ ... ]
>>>>>> Qemu test results:
>>>>>>     total: 96 pass: 69 fail: 27
>>>>>> Failed tests:
>>>>> [ ... ]
>>>>>>     mips:mips_malta_smp_defconfig
>>>>> I bisected this failure to commit bb11cff327e54 ("MIPS: Make smp 
>>>>> CMP, CPS and MT
>>>>> use the new generic IPI functions". Bisect log is attached.
>>>> Thanks for bisecting this. I tested this on a real Malta system but 
>>>> not
>>>> qemu. I'll try to reproduce.
>>>>
>>> I run the tests with only a single CPU core enabled. Maybe that causes
>>> problems with your code ?
>>>
>> I ran another qemu test (this time on mainline) with "-smp 2", but 
>> the only
>> difference is that the image now gets stuck even earlier.
>>
>> Also, I ran another set of bisects, this time with both mips and mips64
>> on mainline (after your patch landed), with the same results.
>>
>> Guenter
>>
>
> OK thanks for the info. The offending commit just enables using quite 
> a few of the newly added code before that. So the problem could be in 
> any of the newly added code.
>
> Unfortunately I can only look at this during my limited time in the 
> evening and I have to setup my system to compile and run this, so I 
> won't be able to get to the bottom of this as fast as I'd like to.
>
> Qais

OK I was up and running faster than I thought I would be. Can you 
confirm that you're hitting a BUG_ON() in mips_smp_ipi_init()?

What I see is that BUG_ON() is hit because we couldn't find an ipidomain 
to allocate the ipis from. The reason of whih is that the qemu malta 
machine doesn't have a GIC though the config is compiled with GIC on. 
Also if I remember correctly qemu malta doesn't really support SMP. I 
think that was the reason I never ran this on qemu.

I'm not sure what's the best way forward here. I can add a check to 
verify gic_present inside this function and return early. Patch attached.

Ralf, thoughts?

Thanks,
Qais

Re: linux-next: Tree for Mar 14 (mips qemu failure bisected)

[Guenter Roeck]

On 03/16/2016 03:17 PM, Qais Yousef wrote:
> On 16/03/2016 20:27, Qais Yousef wrote:
>>
>>
>> On 16/03/2016 13:22, Guenter Roeck wrote:
>>> On Tue, Mar 15, 2016 at 05:17:13PM -0700, Guenter Roeck wrote:
>>>> On Tue, Mar 15, 2016 at 09:55:06PM +0000, Qais Yousef wrote:
>>>>> Hi Guenter,
>>>>>
>>> [ ... ]
>>>>>>> Qemu test results:
>>>>>>>     total: 96 pass: 69 fail: 27
>>>>>>> Failed tests:
>>>>>> [ ... ]
>>>>>>>     mips:mips_malta_smp_defconfig
>>>>>> I bisected this failure to commit bb11cff327e54 ("MIPS: Make smp CMP, CPS and MT
>>>>>> use the new generic IPI functions". Bisect log is attached.
>>>>> Thanks for bisecting this. I tested this on a real Malta system but not
>>>>> qemu. I'll try to reproduce.
>>>>>
>>>> I run the tests with only a single CPU core enabled. Maybe that causes
>>>> problems with your code ?
>>>>
>>> I ran another qemu test (this time on mainline) with "-smp 2", but the only
>>> difference is that the image now gets stuck even earlier.
>>>
>>> Also, I ran another set of bisects, this time with both mips and mips64
>>> on mainline (after your patch landed), with the same results.
>>>
>>> Guenter
>>>
>>
>> OK thanks for the info. The offending commit just enables using quite a few of the newly added code before that. So the problem could be in any of the newly added code.
>>
>> Unfortunately I can only look at this during my limited time in the evening and I have to setup my system to compile and run this, so I won't be able to get to the bottom of this as fast as I'd like to.
>>
>> Qais
>
> OK I was up and running faster than I thought I would be. Can you confirm that you're hitting a BUG_ON() in mips_smp_ipi_init()?
>

Most likely, but mips is one of the qemu emulations which simply hang if there is a crash,
without a log message, and I have not been able to figure out a command line that gives me
the actual crash log.

> What I see is that BUG_ON() is hit because we couldn't find an ipidomain to allocate the ipis from. The reason of whih is that the qemu malta machine doesn't have a GIC though the config is compiled with GIC on. Also if I remember correctly qemu malta doesn't really support SMP. I think that was the reason I never ran this on qemu.
>
Idea is to run the SMP build, not really a multi-core machine.

> I'm not sure what's the best way forward here. I can add a check to verify gic_present inside this function and return early. Patch attached.
>

With your patch I get

WARNING: CPU: 0 PID: 1 at arch/mips/kernel/smp.c:251 mips_smp_ipi_init+0x3c/0x1b0()

and the boot continues.

If Ralf accepts it, feel free to add

Tested-by: Guenter Roeck <linux@roeck-us.net>

though I'll probably adjust my configuration to drop GIC from it (if that is possible).

Thanks a lot for looking into this!

Guenter

Re: linux-next: Tree for Mar 14 (mips qemu failure bisected)

[Guenter Roeck]

On 03/16/2016 03:17 PM, Qais Yousef wrote:
> On 16/03/2016 20:27, Qais Yousef wrote:
>>
>>
>> On 16/03/2016 13:22, Guenter Roeck wrote:
>>> On Tue, Mar 15, 2016 at 05:17:13PM -0700, Guenter Roeck wrote:
>>>> On Tue, Mar 15, 2016 at 09:55:06PM +0000, Qais Yousef wrote:
>>>>> Hi Guenter,
>>>>>
>>> [ ... ]
>>>>>>> Qemu test results:
>>>>>>>     total: 96 pass: 69 fail: 27
>>>>>>> Failed tests:
>>>>>> [ ... ]
>>>>>>>     mips:mips_malta_smp_defconfig
>>>>>> I bisected this failure to commit bb11cff327e54 ("MIPS: Make smp CMP, CPS and MT
>>>>>> use the new generic IPI functions". Bisect log is attached.
>>>>> Thanks for bisecting this. I tested this on a real Malta system but not
>>>>> qemu. I'll try to reproduce.
>>>>>
>>>> I run the tests with only a single CPU core enabled. Maybe that causes
>>>> problems with your code ?
>>>>
>>> I ran another qemu test (this time on mainline) with "-smp 2", but the only
>>> difference is that the image now gets stuck even earlier.
>>>
>>> Also, I ran another set of bisects, this time with both mips and mips64
>>> on mainline (after your patch landed), with the same results.
>>>
>>> Guenter
>>>
>>
>> OK thanks for the info. The offending commit just enables using quite a few of the newly added code before that. So the problem could be in any of the newly added code.
>>
>> Unfortunately I can only look at this during my limited time in the evening and I have to setup my system to compile and run this, so I won't be able to get to the bottom of this as fast as I'd like to.
>>
>> Qais
>
> OK I was up and running faster than I thought I would be. Can you confirm that you're hitting a BUG_ON() in mips_smp_ipi_init()?
>
> What I see is that BUG_ON() is hit because we couldn't find an ipidomain to allocate the ipis from. The reason of whih is that the qemu malta machine doesn't have a GIC though the config is compiled with GIC on. Also if I remember correctly qemu malta doesn't really support SMP. I think that was the reason I never ran this on qemu.

Turns out MIPS_GIC is auto-selected by MIPS_MALTA, so I can not just unconfigure it.
Too bad. That means if your patch isn't accepted, I'll have to drop the mips
SMP build runtime tests, unless you have a better idea.

Thanks,
Guenter

Re: linux-next: Tree for Mar 14 (mips qemu failure bisected)

[Guenter Roeck]

On 03/16/2016 10:25 PM, Qais Yousef wrote:
[ ... ]
>
> Yeah it is assumed that a Malta should always have a GIC and no one got around to fix this in qemu yet.
>
> I can only improve on the patch to do
>
>    if (!ipidomain && nr_cpu_ids == 1)
>            return 0;
>
> Which is more generic way to do it. I think a WARN_ON () would still be useful as SMP without ipis will not work really.
>
> I can send a proper patch if Ralf is OK with that.
>
> I got the console output from qemu by the way. I didn't use your script though.
>

What command line did you use ?

Thanks,
Guenter