(IIO) ADC freeze on am335x processor on 3.14.26ltsi kernel

(IIO) ADC freeze on am335x processor on 3.14.26ltsi kernel

[Yani Dubin]

[-- Attachment #1: Type: text/plain, Size: 3059 bytes --]

Hi,

I have run up against a kernel bug on the dizzy branch where a read of an
ADC channel (IIO bus) on a Sitara (am335x) processor locks up the process
reading the /sys/bus/iio/devices/iio\:device0/in_voltage?_raw node. The
system remains stable, but the locked process does not respond to SIGKILL.
No kernel errors or debugging info in the logs - but I have taken no steps
to improve verbosity. fuser shows the files as being open by the crashed
process.

I did google the issue, but turned up nothing. This is very reproduceable,
and I have done my best to narrow it down on the dizzy timeline. In each
case I have used the same rootfs (built from dizzy tip), but just flashed a
different kernel.

The (yocto) changesets in question are:
36e42c0ddb7a40b3022e9b165560622479f1aa5c  - 3.14.26ltsi exhibits issue
446acfb5a474b23041b46c49732474f2d415160d   - 3.14.24 doesn't build
d7c61053da96d675c3912a45ee448c20ae91721b - 3.14.19 issue not present

The build error in question is "ERROR: Fetcher failure: Unable to find
revision fba2d0cdb745e0f807ce134fd9d1524b7bed9742 in branch meta even from
upstream".

These are 3 adjacent changesets on the dizzy branch - I gather that is as
much as I can narrow it down?

Should I be submitting this bug to yocto bugzilla (since it affects the
stable release, and I am using linux-yocto) or a linux kernel bug tracker /
mailing list? Or is there further information I should be getting (enabling
kernel debugging / increasing verbosity perhaps)? Should I do a build off
the unstable branch for comparison? (to see if this has a newer kernel,
with upstream fix which resolves the issue)

Meanwhile, I thought I would see if anyone else is experiencing this, or
has similar hardware and can see if they are also affected. This is a
custom board, but similar processor-wise to the Beaglebone Black.

Steps to reproduce:
cat /sys/bus/iio/devices/iio\:device0/in_voltage?_raw

The particular drivers in use are:
CONFIG_TI_AM335X_ADC=y
CONFIG_MFD_TI_AM335X_TSCADC=y

Regards,
Yani.

-- 

------------------------------
This email, including any attachments, is only for the intended recipient. 
It is subject to copyright, is confidential and may be the subject of legal 
or other privilege, none of which is waived or lost by reason of this 
transmission.
If you are not an intended recipient, you may not use, disseminate, 
distribute or reproduce such email, any attachments, or any part thereof. 
If you have received a message in error, please notify the sender 
immediately and erase all copies of the message and any attachments.
Unfortunately, we cannot warrant that the email has not been altered or 
corrupted during transmission nor can we guarantee that any email or any 
attachments are free from computer viruses or other conditions which may 
damage or interfere with recipient data, hardware or software. The 
recipient relies upon its own procedures and assumes all risk of use and of 
opening any attachments.
------------------------------

[-- Attachment #2: Type: text/html, Size: 3287 bytes --]

Re: (IIO) ADC freeze on am335x processor on 3.14.26ltsi kernel

[Bruce Ashfield]

[-- Attachment #1: Type: text/plain, Size: 4125 bytes --]

On 15-01-26 11:54 PM, Yani Dubin wrote:
> Hi,
>
> I have run up against a kernel bug on the dizzy branch where a read of 
> an ADC channel (IIO bus) on a Sitara (am335x) processor locks up the 
> process reading the /sys/bus/iio/devices/iio\:device0/in_voltage?_raw 
> node. The system remains stable, but the locked process does not 
> respond to SIGKILL. No kernel errors or debugging info in the logs - 
> but I have taken no steps to improve verbosity. fuser shows the files 
> as being open by the crashed process.
>
> I did google the issue, but turned up nothing. This is very 
> reproduceable, and I have done my best to narrow it down on the dizzy 
> timeline. In each case I have used the same rootfs (built from dizzy 
> tip), but just flashed a different kernel.
>
> The (yocto) changesets in question are:
> 36e42c0ddb7a40b3022e9b165560622479f1aa5c  - 3.14.26ltsi exhibits issue
> 446acfb5a474b23041b46c49732474f2d415160d   - 3.14.24 doesn't build
> d7c61053da96d675c3912a45ee448c20ae91721b - 3.14.19 issue not present
>
> The build error in question is "ERROR: Fetcher failure: Unable to find 
> revision fba2d0cdb745e0f807ce134fd9d1524b7bed9742 in branch meta even 
> from upstream".
>
> These are 3 adjacent changesets on the dizzy branch - I gather that is 
> as much as I can narrow it down?
>
> Should I be submitting this bug to yocto bugzilla (since it affects 
> the stable release, and I am using linux-yocto) or a linux kernel bug 
> tracker / mailing list? Or is there further information I should be 
> getting (enabling kernel debugging / increasing verbosity perhaps)? 
> Should I do a build off the unstable branch for comparison? (to see if 
> this has a newer kernel, with upstream fix which resolves the issue)

We may need to get your help in tracking the problem down, since without
the h/w on hand, it will be difficult to figure out what changed between the
working and non-working revisions.

I have some additional 3.14 updates queued, and they may fix the issue ..
but again, I can't say for sure.

Are you able to bisect the kernel directly ? I only submit SRCREV updates to
the linux-yocto recipes at particular milestones, but within the kernel we
obviously have more granularity.

After you've built the kernel the first time, you can head into devshell 
and
then use the unpacked and patched src tree and do a normal git bisect/build
workflow. You'd need to copy the resulting kernels out to your target, 
but that
would narrow down the bad commit/merge pretty quickly.

Bruce

>
> Meanwhile, I thought I would see if anyone else is experiencing this, 
> or has similar hardware and can see if they are also affected. This is 
> a custom board, but similar processor-wise to the Beaglebone Black.
>
> Steps to reproduce:
> cat /sys/bus/iio/devices/iio\:device0/in_voltage?_raw
>
> The particular drivers in use are:
> CONFIG_TI_AM335X_ADC=y
> CONFIG_MFD_TI_AM335X_TSCADC=y
>
> Regards,
> Yani.
>
> ------------------------------------------------------------------------
> This email, including any attachments, is only for the intended 
> recipient. It is subject to copyright, is confidential and may be the 
> subject of legal or other privilege, none of which is waived or lost 
> by reason of this transmission.
> If you are not an intended recipient, you may not use, disseminate, 
> distribute or reproduce such email, any attachments, or any part 
> thereof. If you have received a message in error, please notify the 
> sender immediately and erase all copies of the message and any 
> attachments.
> Unfortunately, we cannot warrant that the email has not been altered 
> or corrupted during transmission nor can we guarantee that any email 
> or any attachments are free from computer viruses or other conditions 
> which may damage or interfere with recipient data, hardware or 
> software. The recipient relies upon its own procedures and assumes all 
> risk of use and of opening any attachments.
> ------------------------------------------------------------------------
>
>


[-- Attachment #2: Type: text/html, Size: 5962 bytes --]

Re: (IIO) ADC freeze on am335x processor on 3.14.26ltsi kernel

[Yani Dubin]

[-- Attachment #1: Type: text/plain, Size: 2917 bytes --]

Hi Bruce,

On 28 January 2015 at 04:08, Bruce Ashfield <bruce.ashfield@windriver.com>
wrote:

>  On 15-01-26 11:54 PM, Yani Dubin wrote:
>
> These are 3 adjacent changesets on the dizzy branch - I gather that is as
> much as I can narrow it down?
>
>  We may need to get your help in tracking the problem down, since without
> the h/w on hand, it will be difficult to figure out what changed between
> the
> working and non-working revisions.
>
> I have some additional 3.14 updates queued, and they may fix the issue ..
> but again, I can't say for sure.
>
> Are you able to bisect the kernel directly ? I only submit SRCREV updates
> to
> the linux-yocto recipes at particular milestones, but within the kernel we
> obviously have more granularity.
>
> After you've built the kernel the first time, you can head into devshell
> and
> then use the unpacked and patched src tree and do a normal git bisect/build
> workflow. You'd need to copy the resulting kernels out to your target, but
> that
> would narrow down the bad commit/merge pretty quickly.
>

Bisecting the 1800+ changesets between these points yielded a culprit:
[bb01842184bc89666819ee7dceb78c61505036ed] mfd: ti_am335x_tscadc: Fix TSC
operation after ADC continouous mode

This is a two line changeset. I have confirmed that reverting the second
change against the previously broken version fixes it for me. So I now have
a patch I can apply locally until such time as this is fixed upstream.

Since this is my first time reporting a kernel bug, and I now know exactly
what I am looking for can you give me any pointers on where/how to search
to see if an upstream fix exists already?

Thank you for your help, and a welcome introduction to both devshell (I
used to do kernel reconfigure/compile the hard way) and git bisect - I can
see these are going to be very handy in the future.

Thanks again,
Yani.

-- 

------------------------------
This email, including any attachments, is only for the intended recipient. 
It is subject to copyright, is confidential and may be the subject of legal 
or other privilege, none of which is waived or lost by reason of this 
transmission.
If you are not an intended recipient, you may not use, disseminate, 
distribute or reproduce such email, any attachments, or any part thereof. 
If you have received a message in error, please notify the sender 
immediately and erase all copies of the message and any attachments.
Unfortunately, we cannot warrant that the email has not been altered or 
corrupted during transmission nor can we guarantee that any email or any 
attachments are free from computer viruses or other conditions which may 
damage or interfere with recipient data, hardware or software. The 
recipient relies upon its own procedures and assumes all risk of use and of 
opening any attachments.
------------------------------

[-- Attachment #2: Type: text/html, Size: 3722 bytes --]

Re: (IIO) ADC freeze on am335x processor on 3.14.26ltsi kernel

[Yani Dubin]

[-- Attachment #1: Type: text/plain, Size: 2017 bytes --]

On 28 January 2015 at 12:19, Yani Dubin <yani.dubin@taitradio.com> wrote:

> Since this is my first time reporting a kernel bug, and I now know exactly
> what I am looking for can you give me any pointers on where/how to search
> to see if an upstream fix exists already?
>

I fetched the latest 3.14.y from linux-stable at git.kernel.org and saw no
further changes to this driver, nor anything relevant on the 3.18.y branch
(git log on affected file). I haven't looked outside the stable source tree.

I have assumed that a fix would touch the driver itself. I do not know what
a proper fix would actually look like - presumably my workaround would
break whatever the changeset was fixing for those using the TSC part of the
driver, much the way their fix broke my use of the driver.

bugzilla.kernel.org states that it is for submitting issues with mainline
rather than distribution kernels. Would they consider linux-yocto a
distribution kernel, and I should leave this with you?

Regards,
Yani.

-- 

------------------------------
This email, including any attachments, is only for the intended recipient. 
It is subject to copyright, is confidential and may be the subject of legal 
or other privilege, none of which is waived or lost by reason of this 
transmission.
If you are not an intended recipient, you may not use, disseminate, 
distribute or reproduce such email, any attachments, or any part thereof. 
If you have received a message in error, please notify the sender 
immediately and erase all copies of the message and any attachments.
Unfortunately, we cannot warrant that the email has not been altered or 
corrupted during transmission nor can we guarantee that any email or any 
attachments are free from computer viruses or other conditions which may 
damage or interfere with recipient data, hardware or software. The 
recipient relies upon its own procedures and assumes all risk of use and of 
opening any attachments.
------------------------------

[-- Attachment #2: Type: text/html, Size: 2598 bytes --]

Re: (IIO) ADC freeze on am335x processor on 3.14.26ltsi kernel

[Bruce Ashfield]

On 2015-01-27 9:13 PM, Yani Dubin wrote:
> On 28 January 2015 at 12:19, Yani Dubin <yani.dubin@taitradio.com
> <mailto:yani.dubin@taitradio.com>> wrote:
>
>     Since this is my first time reporting a kernel bug, and I now know
>     exactly what I am looking for can you give me any pointers on
>     where/how to search to see if an upstream fix exists already?
>
>
> I fetched the latest 3.14.y from linux-stable at git.kernel.org
> <http://git.kernel.org> and saw no further changes to this driver, nor
> anything relevant on the 3.18.y branch (git log on affected file). I
> haven't looked outside the stable source tree.
>
> I have assumed that a fix would touch the driver itself. I do not know
> what a proper fix would actually look like - presumably my workaround
> would break whatever the changeset was fixing for those using the TSC
> part of the driver, much the way their fix broke my use of the driver.
>
> bugzilla.kernel.org <http://bugzilla.kernel.org> states that it is for
> submitting issues with mainline rather than distribution kernels. Would
> they consider linux-yocto a distribution kernel, and I should leave this
> with you?

They would consider it a distribution kernel, but if you can show the
issue in the same 3.14.y -stable kernel, you can report it via that
avenue.

Alternatively, it is worth firing an email to the arm-kernel mailing
list, to see if anyone else has seen this, or has a comment on the
problematic commit.

I'm in the same situation as before if it is left with me .. I don't have
the h/w to reproduce the issue, so I can't effectively fix it.

Bruce

>
> Regards,
> Yani.
>
> ------------------------------------------------------------------------
> This email, including any attachments, is only for the intended
> recipient. It is subject to copyright, is confidential and may be the
> subject of legal or other privilege, none of which is waived or lost by
> reason of this transmission.
> If you are not an intended recipient, you may not use, disseminate,
> distribute or reproduce such email, any attachments, or any part
> thereof. If you have received a message in error, please notify the
> sender immediately and erase all copies of the message and any attachments.
> Unfortunately, we cannot warrant that the email has not been altered or
> corrupted during transmission nor can we guarantee that any email or any
> attachments are free from computer viruses or other conditions which may
> damage or interfere with recipient data, hardware or software. The
> recipient relies upon its own procedures and assumes all risk of use and
> of opening any attachments.
> ------------------------------------------------------------------------