From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Sun, 25 Mar 2018 08:19:04 -0700 From: Tony Lindgren To: Tero Kristo Cc: Christophe Lyon , Stephen Boyd , Jerome Brunet , Michael Turquette , Shawn Lin , Arnd Bergmann , Jyri Sarha , Thorsten Leemhuis , linux-omap , Linux ARM , linux-clk Subject: Re: Possible kernel bug in torvalds/linux/master Message-ID: <20180325151904.GA5700@atomide.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 In-Reply-To: List-ID: Hi, * Arnd Bergmann [180325 13:30]: > On Sun, Mar 25, 2018 at 3:03 PM, Christophe Lyon > wrote: > > Hi Arnd, > > > > We have a Jenkins jobs that builds the kernel from torvalds/linux > > master branch mutli_v7 defconfig every day, using our last GCC release > > (7.2-2017-11), and boots a beaglebone-black board. > > > > Last week it started to fail, I first suspected a Lava problem, but > > the job now fails every time, and Remi Duraffort from the Lava team > > thinks it's really a kernel problem. > > > > Is this something you are interested in investigating? Or should we > > switch to another "less-edge" branch? > > > > The last successful run: > > https://ci.linaro.org/job/tcwg-buildapp/app=linux+multi_v7,label=tcwg-x86_64-build,target=arm-linux-gnueabihf/75/ > > The next one failed: > > https://ci.linaro.org/job/tcwg-buildapp/app=linux+multi_v7,label=tcwg-x86_64-build,target=arm-linux-gnueabihf/76 > > > > Build 75 was with this kernel commit: > > Merge branch 'for-4.16-fixes' > > 1b5f3ba415fe4cf8b8b39c8d104ed44cde330658 > > > > Build 76 was with: > > Merge tag 'clk-fixes-for-linus' > > 3215b9d57a2c75c4305a3956ca303d7004485200 > > Hi Christophe, > > This branch is certainly the right one to test, thanks for the report! > From looking at the output above, it seems that the kernel no longer > boots at all, and fails to even print any messages. Between the > two runs, I see the following commits: > > 3215b9d57a2c Merge tag 'clk-fixes-for-linus' of > git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux > 303851e14a8f Merge tag 'for-linus' of > git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma > 76c0b6a36a12 Merge tag 'scsi-fixes' of > git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi > 645102eac15e Merge tag 'nfsd-4.16-1' of git://linux-nfs.org/~bfields/linux > 32d43cd391ba kvm/x86: fix icebp instruction handling > e8980d67d601 RDMA/ucma: Ensure that CM_ID exists prior to access it > 68ef3bc31664 nfsd: remove blocked locks on client teardown > 80cf79ae4f68 RDMA/verbs: Remove restrack entry from XRCD structure > ed65a4dc2208 RDMA/ucma: Fix use-after-free access in ucma_close > 7997f3b2df75 clk: bcm2835: Protect sections updating shared registers > 49012d1bf5f7 clk: bcm2835: Fix ana->maskX definitions > 2975d5de6428 RDMA/ucma: Check AF family prior resolving address > 8a53fc511c5e clk: aspeed: Prevent reset if clock is enabled > d90c76bb6112 clk: aspeed: Fix is_enabled for certain clocks > bd8602ca42f6 infiniband: bnxt_re: use BIT_ULL() for 64-bit bit masks > 5388a508479d infiniband: qplib_fp: fix pointer cast > 42cea83f9524 IB/mlx5: Fix cleanup order on unload > 0c81ffc60d52 RDMA/ucma: Don't allow join attempts for unsupported AF family > 7688f2c3bbf5 RDMA/ucma: Fix access to non-initialized CM_ID object > 9dea9a2ff61c RDMA/core: Do not use invalid destination in determining port reuse > f3f134f5260a RDMA/mlx5: Fix crash while accessing garbage pointer and > freed memory > c2b37f76485f IB/mlx5: Fix integer overflows in mlx5_ib_create_srq > 2c292dbb398e IB/mlx5: Fix out-of-bounds read in create_raw_packet_qp_rq > 14bc1dff7427 scsi: qla2xxx: Remove FC_NO_LOOP_ID for FCP and FC-NVMe Discovery > 318aaf34f117 scsi: libsas: defer ata device eh commands to libata > 55c19eee3b47 clk: qcom: msm8916: Fix return value check in > qcom_apcs_msm8916_clk_probe() > 9903e41ae1f5 clk: hisilicon: hi3660：Fix potential NULL dereference in > hi3660_stub_clk_probe() > 56e1ee353943 Merge branch 'clk-helpers' (early part) into clk-fixes > 04bf9ab3359f clk: fix determine rate error with pass-through clock > 91584eb51b47 Merge branch 'clk-phase' into clk-fixes > bd13c6cbd3c0 Merge tag 'ti-clk-fixes-4.16' of > https://github.com/t-kristo/linux-pm into clk-fixes > a88bb86d58ce Merge tag 'clk-imx-fixes-4.16' of > git://git.kernel.org/pub/scm/linux/kernel/git/shawnguo/linux into > clk-fixes > 957a42e8599a Merge tag 'sunxi-clk-fixes-for-4.16' of > https://git.kernel.org/pub/scm/linux/kernel/git/sunxi/linux into > clk-fixes > 99652a469df1 clk: migrate the count of orphaned clocks at init > 7f95beea3608 clk: update cached phase to respect the fact when setting phase > 762790b75210 clk: ti: am43xx: add set-rate-parent support for display > clkctrl clock > c083dc5f3738 clk: ti: am33xx: add set-rate-parent support for display > clkctrl clock > 49159a9dc3da clk: ti: clkctrl: add support for CLK_SET_RATE_PARENT flag > a275b315334d clk: imx51-imx53: Fix UART4/5 registration on i.MX50 and i.MX53 > 5682e268350f clk: sunxi-ng: a31: Fix CLK_OUT_* clock ops > > Out of these, All the interesting ones are clk related: > > 56e1ee353943 Merge branch 'clk-helpers' (early part) into clk-fixes > 04bf9ab3359f clk: fix determine rate error with pass-through clock > 91584eb51b47 Merge branch 'clk-phase' into clk-fixes > bd13c6cbd3c0 Merge tag 'ti-clk-fixes-4.16' of > https://github.com/t-kristo/linux-pm into clk-fixes > 99652a469df1 clk: migrate the count of orphaned clocks at init > 7f95beea3608 clk: update cached phase to respect the fact when setting phase > 762790b75210 clk: ti: am43xx: add set-rate-parent support for display > clkctrl clock > c083dc5f3738 clk: ti: am33xx: add set-rate-parent support for display > clkctrl clock > 49159a9dc3da clk: ti: clkctrl: add support for CLK_SET_RATE_PARENT flag > > I've added the involved parties to Cc. We also see the same thing on > kernelci, where many OMAP based systems now fail to boot, with the > problem starting at the same commit: > > https://kernelci.org/boot/all/job/mainline/branch/master/kernel/v4.16-rc6-431-gbcfc1f455466/ > > It's possible that this has already been debugged and a fix is being worked on, > but I'm not aware of anything, since I have not followed my email > while travelling. I've confirmed that omap2plus_defconfig boots on bbb while multi_v7_defconfig fails to boot with the following: l4_wkup_cm:clk:0010:0: failed to disable Unhandled fault: external abort on non-linefetch (0x1028) at 0xfa30e054 pgd = 4b21228f [fa30e054] *pgd=48211452(bad) Internal error: : 1028 [#1] SMP ARM Modules linked in: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.16.0-rc6-00075-g3215b9d57a2c #709 Hardware name: Generic AM33XX (Flattened Device Tree) PC is at _update_sysc_cache+0x2c/0x88 LR is at _enable+0x19c/0x274 pc : [] lr : [] psr: 40000013 sp : db0adea0 ip : 00000003 fp : 00000000 r10: c144997c r9 : 00000157 r8 : 00000003 r7 : c151d30c r6 : 00000000 r5 : c1678ef4 r4 : c151b2f0 r3 : fa30e054 r2 : c151b360 r1 : 00000054 r0 : c151b2f0 Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none Control: 10c5387d Table: 80204019 DAC: 00000051 Process swapper/0 (pid: 1, stack limit = 0x2ddf0754) Stack: (0xdb0adea0 to 0xdb0ae000) dea0: c151b2f0 c032afc8 00000000 a0000013 c1504c48 c151b2f0 c151b314 c1504c48 dec0: c151b328 c1311c78 a0000013 c0c15ec4 00000011 edaa6d91 c131297c c151b2f0 dee0: c150ce28 c131297c ffffe000 c1312a68 c1504c48 00000000 c131297c c0302730 df00: dfdffb06 dfdffafa c1250ecc 00000100 00000157 c0361f34 c124f400 c10cc358 df20: 00000000 00000002 00000002 c10dec28 00000000 c1504c48 c10eeca0 c10dec9c df40: 00000000 dfdffb06 00000000 edaa6d91 00000000 c1677700 c1677700 c13cf824 df60: c13cf83c 00000003 00000157 c144997c 00000000 c1300e2c 00000002 00000002 df80: 00000000 c13005c0 00000000 c0d96788 00000000 00000000 00000000 00000000 dfa0: 00000000 c0d96790 00000000 c03010e8 00000000 00000000 00000000 00000000 dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 d5370d56 dcffd777 [] (_update_sysc_cache) from [] (_enable+0x19c/0x274) [] (_enable) from [] (_setup.part.16+0xd8/0x418) [] (_setup.part.16) from [] (__omap_hwmod_setup_all+0xec/0x100) [] (__omap_hwmod_setup_all) from [] (do_one_initcall+0x54/0x18c) [] (do_one_initcall) from [] (kernel_init_freeable+0x144/0x1d0) [] (kernel_init_freeable) from [] (kernel_init+0x8/0x110) [] (kernel_init) from [] (ret_from_fork+0x14/0x2c) Exception stack(0xdb0adfb0 to 0xdb0adff8) dfa0: 00000000 00000000 00000000 00000000 dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 Code: e31c0c01 e5903048 e0833001 1a00000a (e5933000) Tero, it might be some timing related clock issue? Regards, Tony From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tony Lindgren Subject: Re: Possible kernel bug in torvalds/linux/master Date: Sun, 25 Mar 2018 08:19:04 -0700 Message-ID: <20180325151904.GA5700@atomide.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=m.gmane.org@lists.infradead.org To: Tero Kristo Cc: Arnd Bergmann , Stephen Boyd , Shawn Lin , Jyri Sarha , linux-clk , Michael Turquette , Thorsten Leemhuis , linux-omap , Christophe Lyon , Linux ARM , Jerome Brunet List-Id: linux-omap@vger.kernel.org SGksCgoqIEFybmQgQmVyZ21hbm4gPGFybmRAYXJuZGIuZGU+IFsxODAzMjUgMTM6MzBdOgo+IE9u IFN1biwgTWFyIDI1LCAyMDE4IGF0IDM6MDMgUE0sIENocmlzdG9waGUgTHlvbgo+IDxjaHJpc3Rv cGhlLmx5b25AbGluYXJvLm9yZz4gd3JvdGU6Cj4gPiBIaSBBcm5kLAo+ID4KPiA+IFdlIGhhdmUg YSBKZW5raW5zIGpvYnMgdGhhdCBidWlsZHMgdGhlIGtlcm5lbCBmcm9tIHRvcnZhbGRzL2xpbnV4 Cj4gPiBtYXN0ZXIgYnJhbmNoIG11dGxpX3Y3IGRlZmNvbmZpZyBldmVyeSBkYXksIHVzaW5nIG91 ciBsYXN0IEdDQyByZWxlYXNlCj4gPiAoNy4yLTIwMTctMTEpLCBhbmQgYm9vdHMgYSBiZWFnbGVi b25lLWJsYWNrIGJvYXJkLgo+ID4KPiA+IExhc3Qgd2VlayBpdCBzdGFydGVkIHRvIGZhaWwsIEkg Zmlyc3Qgc3VzcGVjdGVkIGEgTGF2YSBwcm9ibGVtLCBidXQKPiA+IHRoZSBqb2Igbm93IGZhaWxz IGV2ZXJ5IHRpbWUsIGFuZCBSZW1pIER1cmFmZm9ydCBmcm9tIHRoZSBMYXZhIHRlYW0KPiA+IHRo aW5rcyBpdCdzIHJlYWxseSBhIGtlcm5lbCBwcm9ibGVtLgo+ID4KPiA+IElzIHRoaXMgc29tZXRo aW5nIHlvdSBhcmUgaW50ZXJlc3RlZCBpbiBpbnZlc3RpZ2F0aW5nPyBPciBzaG91bGQgd2UKPiA+ IHN3aXRjaCB0byBhbm90aGVyICJsZXNzLWVkZ2UiIGJyYW5jaD8KPiA+Cj4gPiBUaGUgbGFzdCBz dWNjZXNzZnVsIHJ1bjoKPiA+IGh0dHBzOi8vY2kubGluYXJvLm9yZy9qb2IvdGN3Zy1idWlsZGFw cC9hcHA9bGludXgrbXVsdGlfdjcsbGFiZWw9dGN3Zy14ODZfNjQtYnVpbGQsdGFyZ2V0PWFybS1s aW51eC1nbnVlYWJpaGYvNzUvCj4gPiBUaGUgbmV4dCBvbmUgZmFpbGVkOgo+ID4gaHR0cHM6Ly9j aS5saW5hcm8ub3JnL2pvYi90Y3dnLWJ1aWxkYXBwL2FwcD1saW51eCttdWx0aV92NyxsYWJlbD10 Y3dnLXg4Nl82NC1idWlsZCx0YXJnZXQ9YXJtLWxpbnV4LWdudWVhYmloZi83Ngo+ID4KPiA+IEJ1 aWxkIDc1IHdhcyB3aXRoIHRoaXMga2VybmVsIGNvbW1pdDoKPiA+IE1lcmdlIGJyYW5jaCAnZm9y LTQuMTYtZml4ZXMnCj4gPiAxYjVmM2JhNDE1ZmU0Y2Y4YjhiMzljOGQxMDRlZDQ0Y2RlMzMwNjU4 Cj4gPgo+ID4gQnVpbGQgNzYgd2FzIHdpdGg6Cj4gPiBNZXJnZSB0YWcgJ2Nsay1maXhlcy1mb3It bGludXMnCj4gPiAzMjE1YjlkNTdhMmM3NWM0MzA1YTM5NTZjYTMwM2Q3MDA0NDg1MjAwCj4gCj4g SGkgQ2hyaXN0b3BoZSwKPiAKPiBUaGlzIGJyYW5jaCBpcyBjZXJ0YWlubHkgdGhlIHJpZ2h0IG9u ZSB0byB0ZXN0LCB0aGFua3MgZm9yIHRoZSByZXBvcnQhCj4gRnJvbSBsb29raW5nIGF0IHRoZSBv dXRwdXQgYWJvdmUsIGl0IHNlZW1zIHRoYXQgdGhlIGtlcm5lbCBubyBsb25nZXIKPiBib290cyBh dCBhbGwsIGFuZCBmYWlscyB0byBldmVuIHByaW50IGFueSBtZXNzYWdlcy4gQmV0d2VlbiB0aGUK PiB0d28gcnVucywgSSBzZWUgdGhlIGZvbGxvd2luZyBjb21taXRzOgo+IAo+IDMyMTViOWQ1N2Ey YyBNZXJnZSB0YWcgJ2Nsay1maXhlcy1mb3ItbGludXMnIG9mCj4gZ2l0Oi8vZ2l0Lmtlcm5lbC5v cmcvcHViL3NjbS9saW51eC9rZXJuZWwvZ2l0L2Nsay9saW51eAo+IDMwMzg1MWUxNGE4ZiBNZXJn ZSB0YWcgJ2Zvci1saW51cycgb2YKPiBnaXQ6Ly9naXQua2VybmVsLm9yZy9wdWIvc2NtL2xpbnV4 L2tlcm5lbC9naXQvcmRtYS9yZG1hCj4gNzZjMGI2YTM2YTEyIE1lcmdlIHRhZyAnc2NzaS1maXhl cycgb2YKPiBnaXQ6Ly9naXQua2VybmVsLm9yZy9wdWIvc2NtL2xpbnV4L2tlcm5lbC9naXQvamVq Yi9zY3NpCj4gNjQ1MTAyZWFjMTVlIE1lcmdlIHRhZyAnbmZzZC00LjE2LTEnIG9mIGdpdDovL2xp bnV4LW5mcy5vcmcvfmJmaWVsZHMvbGludXgKPiAzMmQ0M2NkMzkxYmEga3ZtL3g4NjogZml4IGlj ZWJwIGluc3RydWN0aW9uIGhhbmRsaW5nCj4gZTg5ODBkNjdkNjAxIFJETUEvdWNtYTogRW5zdXJl IHRoYXQgQ01fSUQgZXhpc3RzIHByaW9yIHRvIGFjY2VzcyBpdAo+IDY4ZWYzYmMzMTY2NCBuZnNk OiByZW1vdmUgYmxvY2tlZCBsb2NrcyBvbiBjbGllbnQgdGVhcmRvd24KPiA4MGNmNzlhZTRmNjgg UkRNQS92ZXJiczogUmVtb3ZlIHJlc3RyYWNrIGVudHJ5IGZyb20gWFJDRCBzdHJ1Y3R1cmUKPiBl ZDY1YTRkYzIyMDggUkRNQS91Y21hOiBGaXggdXNlLWFmdGVyLWZyZWUgYWNjZXNzIGluIHVjbWFf Y2xvc2UKPiA3OTk3ZjNiMmRmNzUgY2xrOiBiY20yODM1OiBQcm90ZWN0IHNlY3Rpb25zIHVwZGF0 aW5nIHNoYXJlZCByZWdpc3RlcnMKPiA0OTAxMmQxYmY1ZjcgY2xrOiBiY20yODM1OiBGaXggYW5h LT5tYXNrWCBkZWZpbml0aW9ucwo+IDI5NzVkNWRlNjQyOCBSRE1BL3VjbWE6IENoZWNrIEFGIGZh bWlseSBwcmlvciByZXNvbHZpbmcgYWRkcmVzcwo+IDhhNTNmYzUxMWM1ZSBjbGs6IGFzcGVlZDog UHJldmVudCByZXNldCBpZiBjbG9jayBpcyBlbmFibGVkCj4gZDkwYzc2YmI2MTEyIGNsazogYXNw ZWVkOiBGaXggaXNfZW5hYmxlZCBmb3IgY2VydGFpbiBjbG9ja3MKPiBiZDg2MDJjYTQyZjYgaW5m aW5pYmFuZDogYm54dF9yZTogdXNlIEJJVF9VTEwoKSBmb3IgNjQtYml0IGJpdCBtYXNrcwo+IDUz ODhhNTA4NDc5ZCBpbmZpbmliYW5kOiBxcGxpYl9mcDogZml4IHBvaW50ZXIgY2FzdAo+IDQyY2Vh ODNmOTUyNCBJQi9tbHg1OiBGaXggY2xlYW51cCBvcmRlciBvbiB1bmxvYWQKPiAwYzgxZmZjNjBk NTIgUkRNQS91Y21hOiBEb24ndCBhbGxvdyBqb2luIGF0dGVtcHRzIGZvciB1bnN1cHBvcnRlZCBB RiBmYW1pbHkKPiA3Njg4ZjJjM2JiZjUgUkRNQS91Y21hOiBGaXggYWNjZXNzIHRvIG5vbi1pbml0 aWFsaXplZCBDTV9JRCBvYmplY3QKPiA5ZGVhOWEyZmY2MWMgUkRNQS9jb3JlOiBEbyBub3QgdXNl IGludmFsaWQgZGVzdGluYXRpb24gaW4gZGV0ZXJtaW5pbmcgcG9ydCByZXVzZQo+IGYzZjEzNGY1 MjYwYSBSRE1BL21seDU6IEZpeCBjcmFzaCB3aGlsZSBhY2Nlc3NpbmcgZ2FyYmFnZSBwb2ludGVy IGFuZAo+IGZyZWVkIG1lbW9yeQo+IGMyYjM3Zjc2NDg1ZiBJQi9tbHg1OiBGaXggaW50ZWdlciBv dmVyZmxvd3MgaW4gbWx4NV9pYl9jcmVhdGVfc3JxCj4gMmMyOTJkYmIzOThlIElCL21seDU6IEZp eCBvdXQtb2YtYm91bmRzIHJlYWQgaW4gY3JlYXRlX3Jhd19wYWNrZXRfcXBfcnEKPiAxNGJjMWRm Zjc0Mjcgc2NzaTogcWxhMnh4eDogUmVtb3ZlIEZDX05PX0xPT1BfSUQgZm9yIEZDUCBhbmQgRkMt TlZNZSBEaXNjb3ZlcnkKPiAzMThhYWYzNGYxMTcgc2NzaTogbGlic2FzOiBkZWZlciBhdGEgZGV2 aWNlIGVoIGNvbW1hbmRzIHRvIGxpYmF0YQo+IDU1YzE5ZWVlM2I0NyBjbGs6IHFjb206IG1zbTg5 MTY6IEZpeCByZXR1cm4gdmFsdWUgY2hlY2sgaW4KPiBxY29tX2FwY3NfbXNtODkxNl9jbGtfcHJv YmUoKQo+IDk5MDNlNDFhZTFmNSBjbGs6IGhpc2lsaWNvbjogaGkzNjYw77yaRml4IHBvdGVudGlh bCBOVUxMIGRlcmVmZXJlbmNlIGluCj4gaGkzNjYwX3N0dWJfY2xrX3Byb2JlKCkKPiA1NmUxZWUz NTM5NDMgTWVyZ2UgYnJhbmNoICdjbGstaGVscGVycycgKGVhcmx5IHBhcnQpIGludG8gY2xrLWZp eGVzCj4gMDRiZjlhYjMzNTlmIGNsazogZml4IGRldGVybWluZSByYXRlIGVycm9yIHdpdGggcGFz cy10aHJvdWdoIGNsb2NrCj4gOTE1ODRlYjUxYjQ3IE1lcmdlIGJyYW5jaCAnY2xrLXBoYXNlJyBp bnRvIGNsay1maXhlcwo+IGJkMTNjNmNiZDNjMCBNZXJnZSB0YWcgJ3RpLWNsay1maXhlcy00LjE2 JyBvZgo+IGh0dHBzOi8vZ2l0aHViLmNvbS90LWtyaXN0by9saW51eC1wbSBpbnRvIGNsay1maXhl cwo+IGE4OGJiODZkNThjZSBNZXJnZSB0YWcgJ2Nsay1pbXgtZml4ZXMtNC4xNicgb2YKPiBnaXQ6 Ly9naXQua2VybmVsLm9yZy9wdWIvc2NtL2xpbnV4L2tlcm5lbC9naXQvc2hhd25ndW8vbGludXgg aW50bwo+IGNsay1maXhlcwo+IDk1N2E0MmU4NTk5YSBNZXJnZSB0YWcgJ3N1bnhpLWNsay1maXhl cy1mb3ItNC4xNicgb2YKPiBodHRwczovL2dpdC5rZXJuZWwub3JnL3B1Yi9zY20vbGludXgva2Vy bmVsL2dpdC9zdW54aS9saW51eCBpbnRvCj4gY2xrLWZpeGVzCj4gOTk2NTJhNDY5ZGYxIGNsazog bWlncmF0ZSB0aGUgY291bnQgb2Ygb3JwaGFuZWQgY2xvY2tzIGF0IGluaXQKPiA3Zjk1YmVlYTM2 MDggY2xrOiB1cGRhdGUgY2FjaGVkIHBoYXNlIHRvIHJlc3BlY3QgdGhlIGZhY3Qgd2hlbiBzZXR0 aW5nIHBoYXNlCj4gNzYyNzkwYjc1MjEwIGNsazogdGk6IGFtNDN4eDogYWRkIHNldC1yYXRlLXBh cmVudCBzdXBwb3J0IGZvciBkaXNwbGF5Cj4gY2xrY3RybCBjbG9jawo+IGMwODNkYzVmMzczOCBj bGs6IHRpOiBhbTMzeHg6IGFkZCBzZXQtcmF0ZS1wYXJlbnQgc3VwcG9ydCBmb3IgZGlzcGxheQo+ IGNsa2N0cmwgY2xvY2sKPiA0OTE1OWE5ZGMzZGEgY2xrOiB0aTogY2xrY3RybDogYWRkIHN1cHBv cnQgZm9yIENMS19TRVRfUkFURV9QQVJFTlQgZmxhZwo+IGEyNzViMzE1MzM0ZCBjbGs6IGlteDUx LWlteDUzOiBGaXggVUFSVDQvNSByZWdpc3RyYXRpb24gb24gaS5NWDUwIGFuZCBpLk1YNTMKPiA1 NjgyZTI2ODM1MGYgY2xrOiBzdW54aS1uZzogYTMxOiBGaXggQ0xLX09VVF8qIGNsb2NrIG9wcwo+ IAo+IE91dCBvZiB0aGVzZSwgQWxsIHRoZSBpbnRlcmVzdGluZyBvbmVzIGFyZSBjbGsgcmVsYXRl ZDoKPiAKPiA1NmUxZWUzNTM5NDMgTWVyZ2UgYnJhbmNoICdjbGstaGVscGVycycgKGVhcmx5IHBh cnQpIGludG8gY2xrLWZpeGVzCj4gMDRiZjlhYjMzNTlmIGNsazogZml4IGRldGVybWluZSByYXRl IGVycm9yIHdpdGggcGFzcy10aHJvdWdoIGNsb2NrCj4gOTE1ODRlYjUxYjQ3IE1lcmdlIGJyYW5j aCAnY2xrLXBoYXNlJyBpbnRvIGNsay1maXhlcwo+IGJkMTNjNmNiZDNjMCBNZXJnZSB0YWcgJ3Rp LWNsay1maXhlcy00LjE2JyBvZgo+IGh0dHBzOi8vZ2l0aHViLmNvbS90LWtyaXN0by9saW51eC1w bSBpbnRvIGNsay1maXhlcwo+IDk5NjUyYTQ2OWRmMSBjbGs6IG1pZ3JhdGUgdGhlIGNvdW50IG9m IG9ycGhhbmVkIGNsb2NrcyBhdCBpbml0Cj4gN2Y5NWJlZWEzNjA4IGNsazogdXBkYXRlIGNhY2hl ZCBwaGFzZSB0byByZXNwZWN0IHRoZSBmYWN0IHdoZW4gc2V0dGluZyBwaGFzZQo+IDc2Mjc5MGI3 NTIxMCBjbGs6IHRpOiBhbTQzeHg6IGFkZCBzZXQtcmF0ZS1wYXJlbnQgc3VwcG9ydCBmb3IgZGlz cGxheQo+IGNsa2N0cmwgY2xvY2sKPiBjMDgzZGM1ZjM3MzggY2xrOiB0aTogYW0zM3h4OiBhZGQg c2V0LXJhdGUtcGFyZW50IHN1cHBvcnQgZm9yIGRpc3BsYXkKPiBjbGtjdHJsIGNsb2NrCj4gNDkx NTlhOWRjM2RhIGNsazogdGk6IGNsa2N0cmw6IGFkZCBzdXBwb3J0IGZvciBDTEtfU0VUX1JBVEVf UEFSRU5UIGZsYWcKPiAKPiBJJ3ZlIGFkZGVkIHRoZSBpbnZvbHZlZCBwYXJ0aWVzIHRvIENjLiBX ZSBhbHNvIHNlZSB0aGUgc2FtZSB0aGluZyBvbgo+IGtlcm5lbGNpLCB3aGVyZSBtYW55IE9NQVAg YmFzZWQgc3lzdGVtcyBub3cgZmFpbCB0byBib290LCB3aXRoIHRoZQo+IHByb2JsZW0gc3RhcnRp bmcgYXQgdGhlIHNhbWUgY29tbWl0Ogo+IAo+IGh0dHBzOi8va2VybmVsY2kub3JnL2Jvb3QvYWxs L2pvYi9tYWlubGluZS9icmFuY2gvbWFzdGVyL2tlcm5lbC92NC4xNi1yYzYtNDMxLWdiY2ZjMWY0 NTU0NjYvCj4gCj4gSXQncyBwb3NzaWJsZSB0aGF0IHRoaXMgaGFzIGFscmVhZHkgYmVlbiBkZWJ1 Z2dlZCBhbmQgYSBmaXggaXMgYmVpbmcgd29ya2VkIG9uLAo+IGJ1dCBJJ20gbm90IGF3YXJlIG9m IGFueXRoaW5nLCBzaW5jZSBJIGhhdmUgbm90IGZvbGxvd2VkIG15IGVtYWlsCj4gd2hpbGUgdHJh dmVsbGluZy4KCkkndmUgY29uZmlybWVkIHRoYXQgb21hcDJwbHVzX2RlZmNvbmZpZyBib290cyBv biBiYmIgd2hpbGUKbXVsdGlfdjdfZGVmY29uZmlnIGZhaWxzIHRvIGJvb3Qgd2l0aCB0aGUgZm9s bG93aW5nOgoKbDRfd2t1cF9jbTpjbGs6MDAxMDowOiBmYWlsZWQgdG8gZGlzYWJsZQpVbmhhbmRs ZWQgZmF1bHQ6IGV4dGVybmFsIGFib3J0IG9uIG5vbi1saW5lZmV0Y2ggKDB4MTAyOCkgYXQgMHhm YTMwZTA1NApwZ2QgPSA0YjIxMjI4ZgpbZmEzMGUwNTRdICpwZ2Q9NDgyMTE0NTIoYmFkKQpJbnRl cm5hbCBlcnJvcjogOiAxMDI4IFsjMV0gU01QIEFSTQpNb2R1bGVzIGxpbmtlZCBpbjoKQ1BVOiAw IFBJRDogMSBDb21tOiBzd2FwcGVyLzAgTm90IHRhaW50ZWQgNC4xNi4wLXJjNi0wMDA3NS1nMzIx NWI5ZDU3YTJjICM3MDkKSGFyZHdhcmUgbmFtZTogR2VuZXJpYyBBTTMzWFggKEZsYXR0ZW5lZCBE ZXZpY2UgVHJlZSkKUEMgaXMgYXQgX3VwZGF0ZV9zeXNjX2NhY2hlKzB4MmMvMHg4OApMUiBpcyBh dCBfZW5hYmxlKzB4MTljLzB4Mjc0CnBjIDogWzxjMDMyYTg0ND5dICAgIGxyIDogWzxjMDMyYWZj OD5dICAgIHBzcjogNDAwMDAwMTMKc3AgOiBkYjBhZGVhMCAgaXAgOiAwMDAwMDAwMyAgZnAgOiAw MDAwMDAwMApyMTA6IGMxNDQ5OTdjICByOSA6IDAwMDAwMTU3ICByOCA6IDAwMDAwMDAzCnI3IDog YzE1MWQzMGMgIHI2IDogMDAwMDAwMDAgIHI1IDogYzE2NzhlZjQgIHI0IDogYzE1MWIyZjAKcjMg OiBmYTMwZTA1NCAgcjIgOiBjMTUxYjM2MCAgcjEgOiAwMDAwMDA1NCAgcjAgOiBjMTUxYjJmMApG bGFnczogblpjdiAgSVJRcyBvbiAgRklRcyBvbiAgTW9kZSBTVkNfMzIgIElTQSBBUk0gIFNlZ21l bnQgbm9uZQpDb250cm9sOiAxMGM1Mzg3ZCAgVGFibGU6IDgwMjA0MDE5ICBEQUM6IDAwMDAwMDUx ClByb2Nlc3Mgc3dhcHBlci8wIChwaWQ6IDEsIHN0YWNrIGxpbWl0ID0gMHgyZGRmMDc1NCkKU3Rh Y2s6ICgweGRiMGFkZWEwIHRvIDB4ZGIwYWUwMDApCmRlYTA6IGMxNTFiMmYwIGMwMzJhZmM4IDAw MDAwMDAwIGEwMDAwMDEzIGMxNTA0YzQ4IGMxNTFiMmYwIGMxNTFiMzE0IGMxNTA0YzQ4CmRlYzA6 IGMxNTFiMzI4IGMxMzExYzc4IGEwMDAwMDEzIGMwYzE1ZWM0IDAwMDAwMDExIGVkYWE2ZDkxIGMx MzEyOTdjIGMxNTFiMmYwCmRlZTA6IGMxNTBjZTI4IGMxMzEyOTdjIGZmZmZlMDAwIGMxMzEyYTY4 IGMxNTA0YzQ4IDAwMDAwMDAwIGMxMzEyOTdjIGMwMzAyNzMwCmRmMDA6IGRmZGZmYjA2IGRmZGZm YWZhIGMxMjUwZWNjIDAwMDAwMTAwIDAwMDAwMTU3IGMwMzYxZjM0IGMxMjRmNDAwIGMxMGNjMzU4 CmRmMjA6IDAwMDAwMDAwIDAwMDAwMDAyIDAwMDAwMDAyIGMxMGRlYzI4IDAwMDAwMDAwIGMxNTA0 YzQ4IGMxMGVlY2EwIGMxMGRlYzljCmRmNDA6IDAwMDAwMDAwIGRmZGZmYjA2IDAwMDAwMDAwIGVk YWE2ZDkxIDAwMDAwMDAwIGMxNjc3NzAwIGMxNjc3NzAwIGMxM2NmODI0CmRmNjA6IGMxM2NmODNj IDAwMDAwMDAzIDAwMDAwMTU3IGMxNDQ5OTdjIDAwMDAwMDAwIGMxMzAwZTJjIDAwMDAwMDAyIDAw MDAwMDAyCmRmODA6IDAwMDAwMDAwIGMxMzAwNWMwIDAwMDAwMDAwIGMwZDk2Nzg4IDAwMDAwMDAw IDAwMDAwMDAwIDAwMDAwMDAwIDAwMDAwMDAwCmRmYTA6IDAwMDAwMDAwIGMwZDk2NzkwIDAwMDAw MDAwIGMwMzAxMGU4IDAwMDAwMDAwIDAwMDAwMDAwIDAwMDAwMDAwIDAwMDAwMDAwCmRmYzA6IDAw MDAwMDAwIDAwMDAwMDAwIDAwMDAwMDAwIDAwMDAwMDAwIDAwMDAwMDAwIDAwMDAwMDAwIDAwMDAw MDAwIDAwMDAwMDAwCmRmZTA6IDAwMDAwMDAwIDAwMDAwMDAwIDAwMDAwMDAwIDAwMDAwMDAwIDAw MDAwMDEzIDAwMDAwMDAwIGQ1MzcwZDU2IGRjZmZkNzc3Cls8YzAzMmE4NDQ+XSAoX3VwZGF0ZV9z eXNjX2NhY2hlKSBmcm9tIFs8YzAzMmFmYzg+XSAoX2VuYWJsZSsweDE5Yy8weDI3NCkKWzxjMDMy YWZjOD5dIChfZW5hYmxlKSBmcm9tIFs8YzEzMTFjNzg+XSAoX3NldHVwLnBhcnQuMTYrMHhkOC8w eDQxOCkKWzxjMTMxMWM3OD5dIChfc2V0dXAucGFydC4xNikgZnJvbSBbPGMxMzEyYTY4Pl0gKF9f b21hcF9od21vZF9zZXR1cF9hbGwrMHhlYy8weDEwMCkKWzxjMTMxMmE2OD5dIChfX29tYXBfaHdt b2Rfc2V0dXBfYWxsKSBmcm9tIFs8YzAzMDI3MzA+XSAoZG9fb25lX2luaXRjYWxsKzB4NTQvMHgx OGMpCls8YzAzMDI3MzA+XSAoZG9fb25lX2luaXRjYWxsKSBmcm9tIFs8YzEzMDBlMmM+XSAoa2Vy bmVsX2luaXRfZnJlZWFibGUrMHgxNDQvMHgxZDApCls8YzEzMDBlMmM+XSAoa2VybmVsX2luaXRf ZnJlZWFibGUpIGZyb20gWzxjMGQ5Njc5MD5dIChrZXJuZWxfaW5pdCsweDgvMHgxMTApCls8YzBk OTY3OTA+XSAoa2VybmVsX2luaXQpIGZyb20gWzxjMDMwMTBlOD5dIChyZXRfZnJvbV9mb3JrKzB4 MTQvMHgyYykKRXhjZXB0aW9uIHN0YWNrKDB4ZGIwYWRmYjAgdG8gMHhkYjBhZGZmOCkKZGZhMDog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgMDAwMDAwMDAgMDAwMDAwMDAgMDAw MDAwMDAgMDAwMDAwMDAKZGZjMDogMDAwMDAwMDAgMDAwMDAwMDAgMDAwMDAwMDAgMDAwMDAwMDAg MDAwMDAwMDAgMDAwMDAwMDAgMDAwMDAwMDAgMDAwMDAwMDAKZGZlMDogMDAwMDAwMDAgMDAwMDAw MDAgMDAwMDAwMDAgMDAwMDAwMDAgMDAwMDAwMTMgMDAwMDAwMDAKQ29kZTogZTMxYzBjMDEgZTU5 MDMwNDggZTA4MzMwMDEgMWEwMDAwMGEgKGU1OTMzMDAwKQoKVGVybywgaXQgbWlnaHQgYmUgc29t ZSB0aW1pbmcgcmVsYXRlZCBjbG9jayBpc3N1ZT8KClJlZ2FyZHMsCgpUb255CgpfX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpsaW51eC1hcm0ta2VybmVsIG1h aWxpbmcgbGlzdApsaW51eC1hcm0ta2VybmVsQGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xp c3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1hcm0ta2VybmVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 From: tony@atomide.com (Tony Lindgren) Date: Sun, 25 Mar 2018 08:19:04 -0700 Subject: Possible kernel bug in torvalds/linux/master In-Reply-To: References: Message-ID: <20180325151904.GA5700@atomide.com> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org Hi, * Arnd Bergmann [180325 13:30]: > On Sun, Mar 25, 2018 at 3:03 PM, Christophe Lyon > wrote: > > Hi Arnd, > > > > We have a Jenkins jobs that builds the kernel from torvalds/linux > > master branch mutli_v7 defconfig every day, using our last GCC release > > (7.2-2017-11), and boots a beaglebone-black board. > > > > Last week it started to fail, I first suspected a Lava problem, but > > the job now fails every time, and Remi Duraffort from the Lava team > > thinks it's really a kernel problem. > > > > Is this something you are interested in investigating? Or should we > > switch to another "less-edge" branch? > > > > The last successful run: > > https://ci.linaro.org/job/tcwg-buildapp/app=linux+multi_v7,label=tcwg-x86_64-build,target=arm-linux-gnueabihf/75/ > > The next one failed: > > https://ci.linaro.org/job/tcwg-buildapp/app=linux+multi_v7,label=tcwg-x86_64-build,target=arm-linux-gnueabihf/76 > > > > Build 75 was with this kernel commit: > > Merge branch 'for-4.16-fixes' > > 1b5f3ba415fe4cf8b8b39c8d104ed44cde330658 > > > > Build 76 was with: > > Merge tag 'clk-fixes-for-linus' > > 3215b9d57a2c75c4305a3956ca303d7004485200 > > Hi Christophe, > > This branch is certainly the right one to test, thanks for the report! > From looking at the output above, it seems that the kernel no longer > boots at all, and fails to even print any messages. Between the > two runs, I see the following commits: > > 3215b9d57a2c Merge tag 'clk-fixes-for-linus' of > git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux > 303851e14a8f Merge tag 'for-linus' of > git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma > 76c0b6a36a12 Merge tag 'scsi-fixes' of > git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi > 645102eac15e Merge tag 'nfsd-4.16-1' of git://linux-nfs.org/~bfields/linux > 32d43cd391ba kvm/x86: fix icebp instruction handling > e8980d67d601 RDMA/ucma: Ensure that CM_ID exists prior to access it > 68ef3bc31664 nfsd: remove blocked locks on client teardown > 80cf79ae4f68 RDMA/verbs: Remove restrack entry from XRCD structure > ed65a4dc2208 RDMA/ucma: Fix use-after-free access in ucma_close > 7997f3b2df75 clk: bcm2835: Protect sections updating shared registers > 49012d1bf5f7 clk: bcm2835: Fix ana->maskX definitions > 2975d5de6428 RDMA/ucma: Check AF family prior resolving address > 8a53fc511c5e clk: aspeed: Prevent reset if clock is enabled > d90c76bb6112 clk: aspeed: Fix is_enabled for certain clocks > bd8602ca42f6 infiniband: bnxt_re: use BIT_ULL() for 64-bit bit masks > 5388a508479d infiniband: qplib_fp: fix pointer cast > 42cea83f9524 IB/mlx5: Fix cleanup order on unload > 0c81ffc60d52 RDMA/ucma: Don't allow join attempts for unsupported AF family > 7688f2c3bbf5 RDMA/ucma: Fix access to non-initialized CM_ID object > 9dea9a2ff61c RDMA/core: Do not use invalid destination in determining port reuse > f3f134f5260a RDMA/mlx5: Fix crash while accessing garbage pointer and > freed memory > c2b37f76485f IB/mlx5: Fix integer overflows in mlx5_ib_create_srq > 2c292dbb398e IB/mlx5: Fix out-of-bounds read in create_raw_packet_qp_rq > 14bc1dff7427 scsi: qla2xxx: Remove FC_NO_LOOP_ID for FCP and FC-NVMe Discovery > 318aaf34f117 scsi: libsas: defer ata device eh commands to libata > 55c19eee3b47 clk: qcom: msm8916: Fix return value check in > qcom_apcs_msm8916_clk_probe() > 9903e41ae1f5 clk: hisilicon: hi3660?Fix potential NULL dereference in > hi3660_stub_clk_probe() > 56e1ee353943 Merge branch 'clk-helpers' (early part) into clk-fixes > 04bf9ab3359f clk: fix determine rate error with pass-through clock > 91584eb51b47 Merge branch 'clk-phase' into clk-fixes > bd13c6cbd3c0 Merge tag 'ti-clk-fixes-4.16' of > https://github.com/t-kristo/linux-pm into clk-fixes > a88bb86d58ce Merge tag 'clk-imx-fixes-4.16' of > git://git.kernel.org/pub/scm/linux/kernel/git/shawnguo/linux into > clk-fixes > 957a42e8599a Merge tag 'sunxi-clk-fixes-for-4.16' of > https://git.kernel.org/pub/scm/linux/kernel/git/sunxi/linux into > clk-fixes > 99652a469df1 clk: migrate the count of orphaned clocks at init > 7f95beea3608 clk: update cached phase to respect the fact when setting phase > 762790b75210 clk: ti: am43xx: add set-rate-parent support for display > clkctrl clock > c083dc5f3738 clk: ti: am33xx: add set-rate-parent support for display > clkctrl clock > 49159a9dc3da clk: ti: clkctrl: add support for CLK_SET_RATE_PARENT flag > a275b315334d clk: imx51-imx53: Fix UART4/5 registration on i.MX50 and i.MX53 > 5682e268350f clk: sunxi-ng: a31: Fix CLK_OUT_* clock ops > > Out of these, All the interesting ones are clk related: > > 56e1ee353943 Merge branch 'clk-helpers' (early part) into clk-fixes > 04bf9ab3359f clk: fix determine rate error with pass-through clock > 91584eb51b47 Merge branch 'clk-phase' into clk-fixes > bd13c6cbd3c0 Merge tag 'ti-clk-fixes-4.16' of > https://github.com/t-kristo/linux-pm into clk-fixes > 99652a469df1 clk: migrate the count of orphaned clocks at init > 7f95beea3608 clk: update cached phase to respect the fact when setting phase > 762790b75210 clk: ti: am43xx: add set-rate-parent support for display > clkctrl clock > c083dc5f3738 clk: ti: am33xx: add set-rate-parent support for display > clkctrl clock > 49159a9dc3da clk: ti: clkctrl: add support for CLK_SET_RATE_PARENT flag > > I've added the involved parties to Cc. We also see the same thing on > kernelci, where many OMAP based systems now fail to boot, with the > problem starting at the same commit: > > https://kernelci.org/boot/all/job/mainline/branch/master/kernel/v4.16-rc6-431-gbcfc1f455466/ > > It's possible that this has already been debugged and a fix is being worked on, > but I'm not aware of anything, since I have not followed my email > while travelling. I've confirmed that omap2plus_defconfig boots on bbb while multi_v7_defconfig fails to boot with the following: l4_wkup_cm:clk:0010:0: failed to disable Unhandled fault: external abort on non-linefetch (0x1028) at 0xfa30e054 pgd = 4b21228f [fa30e054] *pgd=48211452(bad) Internal error: : 1028 [#1] SMP ARM Modules linked in: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.16.0-rc6-00075-g3215b9d57a2c #709 Hardware name: Generic AM33XX (Flattened Device Tree) PC is at _update_sysc_cache+0x2c/0x88 LR is at _enable+0x19c/0x274 pc : [] lr : [] psr: 40000013 sp : db0adea0 ip : 00000003 fp : 00000000 r10: c144997c r9 : 00000157 r8 : 00000003 r7 : c151d30c r6 : 00000000 r5 : c1678ef4 r4 : c151b2f0 r3 : fa30e054 r2 : c151b360 r1 : 00000054 r0 : c151b2f0 Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none Control: 10c5387d Table: 80204019 DAC: 00000051 Process swapper/0 (pid: 1, stack limit = 0x2ddf0754) Stack: (0xdb0adea0 to 0xdb0ae000) dea0: c151b2f0 c032afc8 00000000 a0000013 c1504c48 c151b2f0 c151b314 c1504c48 dec0: c151b328 c1311c78 a0000013 c0c15ec4 00000011 edaa6d91 c131297c c151b2f0 dee0: c150ce28 c131297c ffffe000 c1312a68 c1504c48 00000000 c131297c c0302730 df00: dfdffb06 dfdffafa c1250ecc 00000100 00000157 c0361f34 c124f400 c10cc358 df20: 00000000 00000002 00000002 c10dec28 00000000 c1504c48 c10eeca0 c10dec9c df40: 00000000 dfdffb06 00000000 edaa6d91 00000000 c1677700 c1677700 c13cf824 df60: c13cf83c 00000003 00000157 c144997c 00000000 c1300e2c 00000002 00000002 df80: 00000000 c13005c0 00000000 c0d96788 00000000 00000000 00000000 00000000 dfa0: 00000000 c0d96790 00000000 c03010e8 00000000 00000000 00000000 00000000 dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 d5370d56 dcffd777 [] (_update_sysc_cache) from [] (_enable+0x19c/0x274) [] (_enable) from [] (_setup.part.16+0xd8/0x418) [] (_setup.part.16) from [] (__omap_hwmod_setup_all+0xec/0x100) [] (__omap_hwmod_setup_all) from [] (do_one_initcall+0x54/0x18c) [] (do_one_initcall) from [] (kernel_init_freeable+0x144/0x1d0) [] (kernel_init_freeable) from [] (kernel_init+0x8/0x110) [] (kernel_init) from [] (ret_from_fork+0x14/0x2c) Exception stack(0xdb0adfb0 to 0xdb0adff8) dfa0: 00000000 00000000 00000000 00000000 dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 Code: e31c0c01 e5903048 e0833001 1a00000a (e5933000) Tero, it might be some timing related clock issue? Regards, Tony