bit errors on spitz

bit errors on spitz

[Pavel Machek]

Hi!

I'm getting way too many bit errors on spitz, with various
kernels.

It may be tied to network usage (bluetooth or wifi?). It happens even
on AC power.
									Pavel


-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

bit errors on spitz

[Eric Miao]

On Sat, Mar 6, 2010 at 5:27 AM, Pavel Machek <pavel@ucw.cz> wrote:
> Hi!
>
> I'm getting way too many bit errors on spitz, with various
> kernels.
>
> It may be tied to network usage (bluetooth or wifi?). It happens even
> on AC power.

Pavel,

What kind of bit errors? I'm not using any network here on my spitz so
not sure what exactly was happening. Could you paste the dmesg here
so we can help take a look?

- eric

bit errors on spitz

[Pavel Machek]

> On Sat, Mar 6, 2010 at 5:27 AM, Pavel Machek <pavel@ucw.cz> wrote:
> > Hi!
> >
> > I'm getting way too many bit errors on spitz, with various
> > kernels.
> >
> > It may be tied to network usage (bluetooth or wifi?). It happens even
> > on AC power.
> 
> Pavel,
> 
> What kind of bit errors? I'm not using any network here on my spitz so
> not sure what exactly was happening. Could you paste the dmesg here
> so we can help take a look?

dmesg would not be useful, it usually hits user programs. Like... mutt
suddenly displaying , instead of - in the header. Program failing to
start because function printg is not found (it was not exactly
printf->printg,  I don't remember exact symbol), ping complaining
about discarding corrupted packets, etc.

(Or of course, kernel oopsing or not going from suspend at all. But as
even user data are being corrupted, oops is not likely to be
interesting and system is typically not in state to capture it any more.)

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

bit errors on spitz

[Cyril Hrubis]

Hi!
> dmesg would not be useful, it usually hits user programs. Like... mutt
> suddenly displaying , instead of - in the header. Program failing to
> start because function printg is not found (it was not exactly
> printf->printg,  I don't remember exact symbol), ping complaining
> about discarding corrupted packets, etc.
> 
> (Or of course, kernel oopsing or not going from suspend at all. But as
> even user data are being corrupted, oops is not likely to be
> interesting and system is typically not in state to capture it any more.)
>

Well I've seen empty lines when editing file with vim (these that are starting
with blue tilda) in the middle of file. And sometimes programs segfaults for no
good reason. Just today I've run "apt-get update" and got:

symbol lookup error: apt-get: undefined symbol: _ZN16pkgAcquireStatus4StopEv

While the correct symbol seems to be _ZN16pkgAcquireStatus4.

When running 'make' in kernel directory and closing the display sometimes
machine dies and nothing but reset under battery cover helps. I remeber waking
up in the morning, opening the device and reseting the device. And it seems to
be provoked much more by active CF wifi card.

-- 
metan

bit errors on spitz

[Stanislav Brabec]

Cyril Hrubis wrote:

> Well I've seen empty lines when editing file with vim

And I have seen:

- Unreproducible SIGSEGV of gcc (while Wi-Fi connection over CF card was
  running).
- Unreproducible SIGSEGV of opkg (downloading via Wi-Fi connection over
  CF card).
- Unreproducible SIGSEGV of rm (called from find command launched via
  ssh, networking via Wi-Fi connection over CF card).
  (Hint: Tasks above are HDD-intensive.)
- Lost blocks while copying from CF to SD.
- Lost blocks while copying from HDD to SD.
- Lost blocks while copying from CF to USB flash stick.
- And I see display noise while CF Wi-Fi card is active.

These problems appear in all kernels, at least since 2.6.26.

There is no note in the syslog.


________________________________________________________________________
Stanislav Brabec
http://www.penguin.cz/~utx/zaurus

bit errors on spitz

[Cyril Hrubis]

Hi!
> - And I see display noise while CF Wi-Fi card is active.

Forgotten about this one. See for yourself, notice short black vertical lines
flashing randomly.

http://atrey.karlin.mff.cuni.cz/~metan/outgoing/zaurus_sickness.mpg

-- 
metan

bit errors on spitz

[Russell King - ARM Linux]

On Mon, Mar 08, 2010 at 01:16:35PM +0100, Cyril Hrubis wrote:
> Hi!
> > - And I see display noise while CF Wi-Fi card is active.
> 
> Forgotten about this one. See for yourself, notice short black vertical lines
> flashing randomly.

I haven't looked at the video.

Is this display rotated by 90 degrees?

If so, they're actually horizontal lines as far as the display scanning
is concerned - and that tends to suggest that there's insufficient system
bus bandwidth for all the activity taking place, and the LCD controller
is being starved of data.

I've seen similar (described) effects on SA1110 systems in past years
with low clock rates.

Some of the reports suggest that this happens with multiple kernel versions
and is not something new to the latest kernels.  Please confirm when the
problem started.

bit errors on spitz

[Cyril Hrubis]

Hi!
> Is this display rotated by 90 degrees?

Yes it is.

> If so, they're actually horizontal lines as far as the display scanning
> is concerned - and that tends to suggest that there's insufficient system
> bus bandwidth for all the activity taking place, and the LCD controller
> is being starved of data.

Well, when doing 'echo 0 > /sys/class/graphics/fbcon/rotate_all' for 2.6.33
I've got the same problem but the lines are vertical. However 2.6.24 seems to
work in non rotated mode.

> I've seen similar (described) effects on SA1110 systems in past years
> with low clock rates.
> 
> Some of the reports suggest that this happens with multiple kernel versions
> and is not something new to the latest kernels.  Please confirm when the
> problem started.

As far as I can test in rotated mode it happens for kernels from 2.6.24 to
2.6.33 (I haven't older kernel than 2.6.24 that boots on spitz). 

-- 
metan

[URGENT] Re: bit errors on spitz + voipac

[Marek Vasut]

Dne Po 8. b?ezna 2010 13:16:35 Cyril Hrubis napsal(a):
> Hi!
> 
> > - And I see display noise while CF Wi-Fi card is active.
> 
> Forgotten about this one. See for yourself, notice short black vertical
> lines flashing randomly.
> 
> http://atrey.karlin.mff.cuni.cz/~metan/outgoing/zaurus_sickness.mpg

This is not only case of spitz. I've seen LCD image falling apart on pxafb on 
Voipac PXA270 board. The image was like "torn in half and part of it was moved 
to right, the hole between staying white".

This happened exactly when I started doing a DMA transfer from a harddrive 
attached through pata_pxa. It's perfectly replicable. If I disabled DMA and let 
it run only in PIO, the image was fine.

I assume the corruption Pavel was seeing is related. My guess is the problems 
are caused when DMA between the CPU and a companion chip happens. I dunno if the 
DMA controller doesn't have enough power to supply LCD and the companion chip 
with data, but that's one of my guesses.

btw. Adjusting the DMA descriptor length in pata_pxa didn't help.

Guys, we need to investigate this as it seems to cause problems on many places.

Cheers!

[URGENT] Re: bit errors on spitz + voipac

[Eric Miao]

On Thu, Jun 3, 2010 at 8:01 AM, Marek Vasut <marek.vasut@gmail.com> wrote:
> Dne Po 8. b?ezna 2010 13:16:35 Cyril Hrubis napsal(a):
>> Hi!
>>
>> > - And I see display noise while CF Wi-Fi card is active.
>>
>> Forgotten about this one. See for yourself, notice short black vertical
>> lines flashing randomly.
>>
>> http://atrey.karlin.mff.cuni.cz/~metan/outgoing/zaurus_sickness.mpg
>
> This is not only case of spitz. I've seen LCD image falling apart on pxafb on
> Voipac PXA270 board. The image was like "torn in half and part of it was moved
> to right, the hole between staying white".
>
> This happened exactly when I started doing a DMA transfer from a harddrive
> attached through pata_pxa. It's perfectly replicable. If I disabled DMA and let
> it run only in PIO, the image was fine.
>
> I assume the corruption Pavel was seeing is related. My guess is the problems
> are caused when DMA between the CPU and a companion chip happens. I dunno if the
> DMA controller doesn't have enough power to supply LCD and the companion chip
> with data, but that's one of my guesses.
>
> btw. Adjusting the DMA descriptor length in pata_pxa didn't help.
>

If there is a FIFO attached, may need the FIFO status when the error happens.
And a dump of the DMA registers would also be helpful.

bit errors on spitz

[Pavel Machek]

On Mon 2010-03-08 12:48:36, Stanislav Brabec wrote:
> Cyril Hrubis wrote:
> 
> > Well I've seen empty lines when editing file with vim
> 
> And I have seen:
> 
> - Unreproducible SIGSEGV of gcc (while Wi-Fi connection over CF card was
>   running).
> - Unreproducible SIGSEGV of opkg (downloading via Wi-Fi connection over
>   CF card).
> - Unreproducible SIGSEGV of rm (called from find command launched via
>   ssh, networking via Wi-Fi connection over CF card).
>   (Hint: Tasks above are HDD-intensive.)
> - Lost blocks while copying from CF to SD.
> - Lost blocks while copying from HDD to SD.
> - Lost blocks while copying from CF to USB flash stick.
> - And I see display noise while CF Wi-Fi card is active.
> 
> These problems appear in all kernels, at least since 2.6.26.
> 
> There is no note in the syslog.

Interesting, I get memory corruption leading to strange
behaviour. Sometimes echo 3 > /proc/sys/vm/drop_caches helps...
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

bit errors on spitz

[Andy Green]

On 03/08/10 08:25, Somebody in the thread at some point said:
> Hi!
>> dmesg would not be useful, it usually hits user programs. Like... mutt
>> suddenly displaying , instead of - in the header. Program failing to
>> start because function printg is not found (it was not exactly
>> printf->printg,  I don't remember exact symbol), ping complaining
>> about discarding corrupted packets, etc.
>>
>> (Or of course, kernel oopsing or not going from suspend at all. But as
>> even user data are being corrupted, oops is not likely to be
>> interesting and system is typically not in state to capture it any more.)
>>
>
> Well I've seen empty lines when editing file with vim (these that are starting
> with blue tilda) in the middle of file. And sometimes programs segfaults for no
> good reason. Just today I've run "apt-get update" and got:
>
> symbol lookup error: apt-get: undefined symbol: _ZN16pkgAcquireStatus4StopEv
>
> While the correct symbol seems to be _ZN16pkgAcquireStatus4.
>
> When running 'make' in kernel directory and closing the display sometimes
> machine dies and nothing but reset under battery cover helps. I remeber waking
> up in the morning, opening the device and reseting the device. And it seems to
> be provoked much more by active CF wifi card.

I saw very similar failures for a long time on our iMX31 based device. 
Eventually I found a Freescale errata where the RAM inside the USB2 
macrocell started to make single bit errors below 1.38V Vcore; ours was 
1.4V at that time but dipped on CPU load.

I cranked up the Vcore to 1.6V and that solved it, we also added some 
ceramic caps to Vcore to help with the dips.

So it might be worth looking at PMU arrangements for Vcore level / look 
for dips with a 'scope (despite this isn't an iMX31).

A characteristic of it was it never caused kernel issues, since the 
kernel didn't come over USB.  It only ever caused troubles on userspace 
stuff.

-Andy

bit errors on spitz

[Stanislav Brabec]

Andy Green wrote:

> I saw very similar failures for a long time on our iMX31 based device. 
> Eventually I found a Freescale errata where the RAM inside the USB2 
> macrocell started to make single bit errors below 1.38V Vcore; ours was 
> 1.4V at that time but dipped on CPU load.

Good tip. It seems that nobody ported driver for the voltage control
chip ISL6271 from 2.4 kernel, and bootloader probably does not set
correct values.

Datasheet:
http://www.penguin.cz/~utx/zaurus/datasheets/power/Xscale/ISL6271.pdf

-- 
Stanislav Brabec
http://www.penguin.cz/~utx/zaurus

bit errors on spitz

[Andy Green]

On 03/11/10 15:42, Somebody in the thread at some point said:
> Andy Green wrote:
>
>> I saw very similar failures for a long time on our iMX31 based device.
>> Eventually I found a Freescale errata where the RAM inside the USB2
>> macrocell started to make single bit errors below 1.38V Vcore; ours was
>> 1.4V at that time but dipped on CPU load.
>
> Good tip. It seems that nobody ported driver for the voltage control
> chip ISL6271 from 2.4 kernel, and bootloader probably does not set
> correct values.
>
> Datasheet:
> http://www.penguin.cz/~utx/zaurus/datasheets/power/Xscale/ISL6271.pdf

Unless there's more to it in the way the zaurus using it that regulator 
isn't programmable digitally.

Reading about your CF Card WLAN related issues they suck down a good 
amount of power when their radio is up, I would definitely suggest 
monitoring with a 'scope the various rails (Vcore, RAM and whatever it 
is the CF Card is powered by) while putting it under load.

-Andy

bit errors on spitz

[Stanislav Brabec]

Andy Green wrote: 
> On 03/11/10 15:42, Somebody in the thread at some point said:
> > Andy Green wrote:
> >
> >> I saw very similar failures for a long time on our iMX31 based device.
> >> Eventually I found a Freescale errata where the RAM inside the USB2
> >> macrocell started to make single bit errors below 1.38V Vcore; ours was
> >> 1.4V at that time but dipped on CPU load.
> >
> > Good tip. It seems that nobody ported driver for the voltage control
> > chip ISL6271 from 2.4 kernel, and bootloader probably does not set
> > correct values.

> Unless there's more to it in the way the zaurus using it that regulator 
> isn't programmable digitally.

OOPS, I made a mistake and linked ISL6721 instead of ISL6271 there.
Now it is fixed:
http://www.penguin.cz/~utx/zaurus/datasheets/power/XScale/ISL6271A.pdf

This one has I2C. It is connected to GPIO 3 (PWR_SCL) and GPIO 4
(PWR_SDA).

It is visible between the black plastic and the large circular coil:
http://www.penguin.cz/~utx/zaurus/teardown#pcbt

> Reading about your CF Card WLAN related issues they suck down a good 
> amount of power when their radio is up, I would definitely suggest 
> monitoring with a 'scope the various rails (Vcore, RAM and whatever it 
> is the CF Card is powered by) while putting it under load.

I guess that Zaurus has a good power design and that voltage should be
constant enough. CF has a dedicated step down (plus 2.8V power detector
(Why so low, if CF standard requres more?)), HDD has a dedicated step
up/down. USB has dedicated step up. Companion chips use dedicated 3.3V
step down. Audio uses dedicated linear regulator. CPU has several
dedicated step downs, CPU 3.3V uses step-up to 5V and then down to 3.3V
(which is shared only with IOPORT).

Nearest common point between CF card power and CPU power is the battery.


________________________________________________________________________

Stanislav Brabec
http://www.penguin.cz/~utx/zaurus

bit errors on spitz

[Andy Green]

On 03/12/10 09:07, Somebody in the thread at some point said:

>> Unless there's more to it in the way the zaurus using it that regulator
>> isn't programmable digitally.
>
> OOPS, I made a mistake and linked ISL6721 instead of ISL6271 there.
> Now it is fixed:
> http://www.penguin.cz/~utx/zaurus/datasheets/power/XScale/ISL6271A.pdf
>
> This one has I2C. It is connected to GPIO 3 (PWR_SCL) and GPIO 4
> (PWR_SDA).

Thanks... that defaults to 1.3V on Vcore if you don't touch it.  I guess 
confirm on the CPU datasheet that it's OK for your selected CPU clock speed.

> I guess that Zaurus has a good power design and that voltage should be
> constant enough. CF has a dedicated step down (plus 2.8V power detector

In that case is the PXA CF driver PIO?  Then it can be the same load on 
Vcore issue in disguise.

-Andy

bit errors on spitz

[Stanislav Brabec]

Andy Green wrote:

> In that case is the PXA CF driver PIO?  Then it can be the same load on 
> Vcore issue in disguise.

There is a proprietary ASIC chip (Sharp Scoop) that handles CF and HDD
access (and also several additional GPIOs):
http://www.penguin.cz/~utx/zaurus/datasheets/ASIC_S1L50752B26B200/412752.PDF

The ASIC runs in dual power mode. HVDD is powered from the 3.3V
dedicated to CF resp. HDD power supply (both may be turned off by the
kernel), LVDD is shared with CPU 3.3V (it is always on).

It seems that there are no other chips connected to the VCC_PLL,
VCC_SRAM and VCC_CORE.

VCC_DRAM is the same 3.3V ans CPU ans ASIC LVDD and also the same as
flash power and flash driver CPLD:
http://www.penguin.cz/~utx/zaurus/datasheets/memory/


________________________________________________________________________
Stanislav Brabec
http://www.penguin.cz/~utx/zaurus

bit errors on spitz

[Andy Green]

On 03/12/10 10:43, Somebody in the thread at some point said:
> Andy Green wrote:
>
>> In that case is the PXA CF driver PIO?  Then it can be the same load on
>> Vcore issue in disguise.
>
> There is a proprietary ASIC chip (Sharp Scoop) that handles CF and HDD
> access (and also several additional GPIOs):
> http://www.penguin.cz/~utx/zaurus/datasheets/ASIC_S1L50752B26B200/412752.PDF

Right but not thinking about its power arrangements, rather the load on 
the CPU itself when it's transferring data to / from CF interface (via 
this ASIC).

If the ASIC has bus master DMA and that's used by the driver then fair 
enough, otherwise if it is done by PIO in the driver "while using CF" 
(as mentioned in most symptoms) becomes the same as saying "during 100% 
load on CPU" which is what leads to dents in Vcore and potential 
instability by that same Vcore path.

-Andy

bit errors on spitz

[Pavel Machek]

On Fri 2010-03-12 10:07:04, Stanislav Brabec wrote:
> Andy Green wrote: 
> > On 03/11/10 15:42, Somebody in the thread at some point said:
> > > Andy Green wrote:
> > >
> > >> I saw very similar failures for a long time on our iMX31 based device.
> > >> Eventually I found a Freescale errata where the RAM inside the USB2
> > >> macrocell started to make single bit errors below 1.38V Vcore; ours was
> > >> 1.4V at that time but dipped on CPU load.
> > >
> > > Good tip. It seems that nobody ported driver for the voltage control
> > > chip ISL6271 from 2.4 kernel, and bootloader probably does not set
> > > correct values.

Are we sure about this one? If we have wrong voltages on various
parts, that kind-off explains it.

Would it be possible to measure (Voltmeter) difference between 2.4
kernel and 2.6 kernel?

								Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

bit errors on spitz

[Stanislav Brabec]

Pavel Machek p??e v Ne 21. 03. 2010 v 21:43 +0100:
> On Fri 2010-03-12 10:07:04, Stanislav Brabec wrote:

> > > > Good tip. It seems that nobody ported driver for the voltage control
> > > > chip ISL6271 from 2.4 kernel, and bootloader probably does not set
> > > > correct values.
> 
> Are we sure about this one? If we have wrong voltages on various
> parts, that kind-off explains it.
> 
> Would it be possible to measure (Voltmeter) difference between 2.4
> kernel and 2.6 kernel?

If you are ready to run Zaurus in dismantled state, then yes. Measure on
the upper pin of the large coil in the center of the
http://www.penguin.cz/~utx/zaurus/pcbt_uc.jpg image or on the testpoint
nearby (probably to the right).

Alternatively, it is possible to write a driver. It is just one byte
write and one byte read via I2C.


________________________________________________________________________
Stanislav Brabec
http://www.penguin.cz/~utx/zaurus

bit errors on spitz

[Pavel Machek]

On Sun 2010-03-21 22:42:33, Stanislav Brabec wrote:
> Pavel Machek p??e v Ne 21. 03. 2010 v 21:43 +0100:
> > On Fri 2010-03-12 10:07:04, Stanislav Brabec wrote:
> 
> > > > > Good tip. It seems that nobody ported driver for the voltage control
> > > > > chip ISL6271 from 2.4 kernel, and bootloader probably does not set
> > > > > correct values.
> > 
> > Are we sure about this one? If we have wrong voltages on various
> > parts, that kind-off explains it.
> > 
> > Would it be possible to measure (Voltmeter) difference between 2.4
> > kernel and 2.6 kernel?
> 
> If you are ready to run Zaurus in dismantled state, then yes. Measure on
> the upper pin of the large coil in the center of the
> http://www.penguin.cz/~utx/zaurus/pcbt_uc.jpg image or on the testpoint
> nearby (probably to the right).

I'm not comfortable dismantling it :-(.

> Alternatively, it is possible to write a driver. It is just one byte
> write and one byte read via I2C.

Do you know what byte it is? That sounds easy enough...

But I have small problem now -- zaurus seems to work mostly fine
now. Does it depend on temperature, or what? Tried mtest,
nothing. Tried compiling kernel, ok... Will try few more times...

								Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

bit errors on spitz

[Stanislav Brabec]

Pavel Machek wrote:

> > Alternatively, it is possible to write a driver. It is just one byte
> > write and one byte read via I2C.
> 
> Do you know what byte it is? That sounds easy enough...

Yes, it should be easy driver. One byte address and then one byte write
or one byte read.

See the datasheet:
http://www.penguin.cz/~utx/zaurus/datasheets/power/XScale/ISL6271A.pdf

Page 11: address
Pages 8 and 9: Data interpretation

-- 
Stanislav Brabec
http://www.penguin.cz/~utx/zaurus