kernel oops in cfi_cmdset_0002.c:do_write_one()...

kernel oops in cfi_cmdset_0002.c:do_write_one()...

[Steven Scholz]

Hi there,

on my embedded PPC system with linux-2.425 I have two flash devices.
The first one (a new MBM29LV652UE) is directly connected to the MPC8xx
and used to store bootloader, kernel and rootfs (jffs2). It's detected
via CFI by do_map_probe("cfi_probe"...).

The second one (MBM29LV800BA) is connected via some glue logic via an FPGA.
Accesses to this one are very slow. For some reason it does not react on the
CFI sequence and thus not detected. So I use do_map_probe("jedec_probe"...)
instead.

...
  Amd/Fujitsu Extended Query Table v1.1 at 0x0040
number of CFI chips: 1
cfi_cmdset_0002: Disabling fast programming due to code brokenness.
init_dab4k_mtd: bank 1, name: DAB4K 0, size:8388608 bytes
Search for id:(04 225b) interleave(1) type(2)
Found: Fujitsu MBM29LV800BA
DAB4K 1: Found 1 x16 devices at 0x0 in 16-bit mode
number of JEDEC chips: 1
init_dab4k_mtd: bank 2, name: DAB4K 1, size:1048576 bytes
DAB4K flash0: Using Static image partition definition
Creating 5 MTD partitions on "DAB4K 0":
0x00120000-0x00800000 : "JFFS2"
0x00000000-0x00030000 : "U-Boot"
0x00030000-0x00040000 : "Environment"
0x00040000-0x00050000 : "FPGA"
0x00050000-0x00120000 : "Kernel"
DAB4K flash1: Using Static image partition definition
Creating 2 MTD partitions on "DAB4K 1":
0x00000000-0x00004000 : "DADSP Boot"
0x00030000-0x00080000 : "Selfstart"

Everything seems fine. "eraseall /dev/mtd6" works. But when I try to
write data to the second flash

	dd if=/dev/urandom of=/dev/mtd6 bs=64k count=1

the kernel crashes with

Warning: DQ5 raised while program operation was in progress, however operation completed OK
Warning: DQ5 raised while program operation was in progress, however operation completed OK
Warning: DQ5 raised while program operation was in progress, however operation completed OK
Waiting for write to complete timed out in do_write_oneword.
Oops: kernel access of bad area, sig: 11
NIP: 0FDF96B8 XER: 00000000 LR: 1006DA30 SP: 7FFFFC90 REGS: c0b31f50 TRAP: 0400
    Not tainted
MSR: 4000d032 EE: 1 PR: 1 FP: 0 ME: 1 IR/DR: 11
TASK = c0b30000[97] 'dd' Last syscall: 4
last math 00000000 last altivec 00000000
GPR00: 1006DAAC 7FFFFC90 00000000 0FEC09E0 7FFFFD40 00010000 1018B000 1017A1B8
GPR08: 0FE0D744 100A0000 00000000 7FFFFC50 42000002 100A6B24 00000000 00000001
GPR16: 00000000 00000000 00000000 00000000 00000001 00000000 00000001 00000000
GPR24: 7FFFFF88 7FFFFF78 00010000 1007C170 7FFFFD40 7FFFFD40 00000005 1007C170
Call backtrace:
0FECCB20 1006DAAC 1006C504 10011A18 10004770 10004360 0FDAFD14
00000000
Kernel panic: Aiee, killing interrupt handler!
In interrupt handler - not syncing
  <0>Rebooting in 10 seconds..

When I increase the timeout in drivers/mtd/chips/cfi_cmdset_0002.c:do_write_oneword()

	/* Polling toggle bits instead of reading back many times
	   This ensures that write operation is really completed,
	   or tells us why it failed. */
	dq6 = CMD(1<<6);
	dq5 = CMD(1<<5);
-	timeo = jiffies + (HZ/1000); /* setting timeout to 1ms for now */
+	timeo = jiffies + (10 * HZ/1000); /* setting timeout to 10ms for now */

then I still get the warnings about "DQ5 raised" but not about the timeout. Thus
the system does not crash. So I suspect that something in the lines:

		} else {
			printk(KERN_WARNING "Waiting for write to complete timed out in do_write_oneword.");

			chip->state = FL_READY;
			wake_up(&chip->wq);
			cfi_spin_unlock(chip->mutex);
			DISABLE_VPP(map);
			ret = -EIO;
		}

causes the crash...

Ideas?

-- 
Steven Scholz

imc Measurement & Control               imc Meßsysteme GmbH
Voltastr. 5                             Voltastr. 5
13355 Berlin                            13355 Berlin
Germany                                 Deutschland

Re: kernel oops in cfi_cmdset_0002.c:do_write_one()...

[Steven Scholz]

Steven Blakeslee wrote:
> I just debugged something like this ealier.  There are timeout values that
> need to be increased for slow access.

But still the system should not crash like this in case of an timeout
(maybe cause by an old flash), don't you think?

I guess somethings wrong with the waitqueue chip->wq ...

> 		} else {
>...
> 			chip->state = FL_READY;
> 			wake_up(&chip->wq);
> 			cfi_spin_unlock(chip->mutex);
> 			DISABLE_VPP(map);
 > ...

Steven Scholz

Re: kernel oops in cfi_cmdset_0002.c:do_write_one()...

[Thayne Harbaugh]

On Wed, 2004-04-07 at 09:43, Steven Scholz wrote:
> Hi there,
> 
> on my embedded PPC system with linux-2.425 I have two flash devices.

> Warning: DQ5 raised while program operation was in progress, however operation completed OK
> Warning: DQ5 raised while program operation was in progress, however operation completed OK
> Warning: DQ5 raised while program operation was in progress, however operation completed OK
> Waiting for write to complete timed out in do_write_oneword.

These error messages are from an ancient version of cfi_cmdset_0002.c.
They come from over-efficient software design that didn't consider the
quirks of how a flash chip works. Have you tried a recent version?

-- 
Thayne Harbaugh
Linux Networx

Re: kernel oops in cfi_cmdset_0002.c:do_write_one()...

[David Vrabel]

> These error messages are from an ancient version of cfi_cmdset_0002.c.
> They come from over-efficient software design that didn't consider the
> quirks of how a flash chip works. Have you tried a recent version?

Be aware that the more recent version from MTD CVS are broken for 
interleaved chips.

http://lists.infradead.org/pipermail/linux-mtd/2004-March/009467.html

David Vrabel

Re: kernel oops in cfi_cmdset_0002.c:do_write_one()...

[Steven Scholz]

Thayne Harbaugh wrote:
> On Wed, 2004-04-07 at 09:43, Steven Scholz wrote:
> 
>>Hi there,
>>
>>on my embedded PPC system with linux-2.425 I have two flash devices.
> 
> 
>>Warning: DQ5 raised while program operation was in progress, however operation completed OK
>>Warning: DQ5 raised while program operation was in progress, however operation completed OK
>>Warning: DQ5 raised while program operation was in progress, however operation completed OK
>>Waiting for write to complete timed out in do_write_oneword.
> 
> 
> These error messages are from an ancient version of cfi_cmdset_0002.c.
> They come from over-efficient software design that didn't consider the
> quirks of how a flash chip works. Have you tried a recent version?

Ehm. I am using a 2.4.25 kernel. I thought the MTD code in there should be 
quiet recent...

How can I check the version of the MTD code in my kernel tree?

-- 
Steven Scholz

imc Measurement & Control               imc Meßsysteme GmbH
Voltastr. 5                             Voltastr. 5
13355 Berlin                            13355 Berlin
Germany                                 Deutschland

Re: kernel oops in cfi_cmdset_0002.c:do_write_one()...

[Steven Scholz]

Thayne Harbaugh wrote:

> On Wed, 2004-04-07 at 09:43, Steven Scholz wrote:
> 
>>Hi there,
>>
>>on my embedded PPC system with linux-2.425 I have two flash devices.
> 
> 
>>Warning: DQ5 raised while program operation was in progress, however operation completed OK
>>Warning: DQ5 raised while program operation was in progress, however operation completed OK
>>Warning: DQ5 raised while program operation was in progress, however operation completed OK
>>Waiting for write to complete timed out in do_write_oneword.
> 
> 
> These error messages are from an ancient version of cfi_cmdset_0002.c.
> They come from over-efficient software design that didn't consider the
> quirks of how a flash chip works. Have you tried a recent version?

I just notice that in my old 2.4.20 kernel tree I have a

	* $Id: cfi_cmdset_0002.c,v 1.67 2003/04/15 22:32:13 thayne Exp $

and in a _new_ 2.4.25 tree I have

	 * $Id: cfi_cmdset_0002.c,v 1.62 2003/01/24 23:30:13 dwmw2 Exp $

How could that be !?!?

Is the MTD code not pushed up into the main kernel sources???

Steven



-- 
Steven Scholz

imc Measurement & Control               imc Meßsysteme GmbH
Voltastr. 5                             Voltastr. 5
13355 Berlin                            13355 Berlin
Germany                                 Deutschland

Re: kernel oops in cfi_cmdset_0002.c:do_write_one()...

[Jörn Engel]

On Thu, 8 April 2004 15:29:09 +0200, Steven Scholz wrote:
> 
> I just notice that in my old 2.4.20 kernel tree I have a
> 
> 	* $Id: cfi_cmdset_0002.c,v 1.67 2003/04/15 22:32:13 thayne Exp $
> 
> and in a _new_ 2.4.25 tree I have
> 
> 	 * $Id: cfi_cmdset_0002.c,v 1.62 2003/01/24 23:30:13 dwmw2 Exp $
> 
> How could that be !?!?
> 
> Is the MTD code not pushed up into the main kernel sources???

It is, whenever David finds the time for it.

Don't blame him, this is the result of using CVS, which is designed
around the assumption of a single central repository.  If current sync
speed is to slow for you, try to make it easier for him to send
patches forward.

Jörn

-- 
Invincibility is in oneself, vulnerability is in the opponent.
-- Sun Tzu

Re: kernel oops in cfi_cmdset_0002.c:do_write_one()...

[Thomas Gleixner]

On Thursday 08 April 2004 16:37, Jörn Engel wrote:
> On Thu, 8 April 2004 15:29:09 +0200, Steven Scholz wrote:
> > I just notice that in my old 2.4.20 kernel tree I have a
> >
> > 	* $Id: cfi_cmdset_0002.c,v 1.67 2003/04/15 22:32:13 thayne Exp $
> >
> > and in a _new_ 2.4.25 tree I have
> >
> > 	 * $Id: cfi_cmdset_0002.c,v 1.62 2003/01/24 23:30:13 dwmw2 Exp $
> >
> > How could that be !?!?
> >
> > Is the MTD code not pushed up into the main kernel sources???
>
> It is, whenever David finds the time for it.
>
> Don't blame him, this is the result of using CVS, which is designed
> around the assumption of a single central repository.  If current sync
> speed is to slow for you, try to make it easier for him to send
> patches forward.

I don't know, if he is really happy, if everybody starts to send patches 
upstream. He does a good job on this and he focussed more on 2.6 in the last 
months. But I'm sure that he can be convinced to bring 2.4 to the same state 
as 2.6 at least. On the other hand we live very well with the MTD patches 
since long, so what :) I never trust the kernel stuff of subsystems, which 
have a active development team :)

-- 
Thomas
________________________________________________________________________
"Free software" is a matter of liberty, not price. To understand the concept,
you should think of "free" as in "free speech,'' not as in "free beer".
________________________________________________________________________
linutronix - competence in embedded & realtime linux
http://www.linutronix.de
mail: tglx@linutronix.de

Re: kernel oops in cfi_cmdset_0002.c:do_write_one()...

[Jörn Engel]

On Thu, 8 April 2004 21:28:23 +0200, Thomas Gleixner wrote:
> On Thursday 08 April 2004 16:37, Jörn Engel wrote:
> >
> > Don't blame him, this is the result of using CVS, which is designed
> > around the assumption of a single central repository.  If current sync
> > speed is to slow for you, try to make it easier for him to send
> > patches forward.
> 
> I don't know, if he is really happy, if everybody starts to send patches 
> upstream. He does a good job on this and he focussed more on 2.6 in the last 
> months. But I'm sure that he can be convinced to bring 2.4 to the same state 
> as 2.6 at least. On the other hand we live very well with the MTD patches 
> since long, so what :) I never trust the kernel stuff of subsystems, which 
> have a active development team :)

That was not what I meant (although it did sound confusing, sorry).

You can prepare patches and send them to David.  Preparing those
patches is stupid and boring work, I guess David likes it as little as
I do.  Then he can see if they are good enough and send them forward.

Or, alternatively, you can accept the current situation as the best
you can get for your money, which is for free.

Jörn

-- 
My second remark is that our intellectual powers are rather geared to
master static relations and that our powers to visualize processes
evolving in time are relatively poorly developed.
-- Edsger W. Dijkstra

Re: kernel oops in cfi_cmdset_0002.c:do_write_one()...

[David Woodhouse]

On Thu, 2004-04-08 at 16:37 +0200, Jörn Engel wrote:
> Don't blame him, this is the result of using CVS, which is designed
> around the assumption of a single central repository.

Well, switching from CVS wouldn't necessarily make any difference. If I
were using something like BitKeeper then there'd still only be one
repository which I actually _care_ about, and the effect would be much
the same. 

I'd be quite interested in setting up a CVS->BK gateway like the ALSA
one though. That might make my life easier.

>   If current sync speed is to slow for you, try to make it easier for
> him to send patches forward.

I've been fairly busy recently; I've updated the code in Linus' tree a
little while ago but it could do with another update. I'm very much
slower to update 2.4 unless it's actually broken. The 2.4 branch is old
and dead now; all development should be on 2.6

The way the update goes is something like this...

1. Wait till the MTD CVS code is actually stable. It's not at the
   moment; cfi_cmdset_0002.c is known to be broken and I can't send it
   off while it's like that.

2. Look at the current version of the kernel I'm updating. For each file
   in drivers/mtd and include/linux/mtd, take the version from CVS with
   matching $Id$ tag -- the version from CVS which was last sent to
   Linus or Marcelo for inclusion.

3. Generate a patch between those versions in my CVS, and the version
   currently in the mainstream kernel -- those are the patches which
   were included in an upstream kernel but didn't go through me. Read
   each change carefully, and see if it's applicable to _current_ CVS
   code. If so, apply it. If not, discard it. If it's discarded because
   the problem was already fixed, that's fine. If it's discarded because
   it's just a bogus change, make a mental note of it; reversion of
   upstream patches needs to be justified (or at least noted) in your
   covering email because it's often done by accident when updating 
   Linus' tree from an external CVS, and it's _very_ much frowned upon.

4. Now you have all the upstream changes merged into current CVS. Drop
   current CVS into a current Linux tree, take a diff between the two. 
   Read it. All of it. Make cosmetic changes in CVS to reduce noise in
   the patch you're planning to send.

5. Read it again. 

6. Build it. Test it if you're feeling conscientious.

7. Read it again. Identify separate change sets from the whole thing;
   'bk citool' is useful for this. Commit each set of changes separately
   with an appropriate changelog. 

8. Stick it in a BK tree somewhere for Linus or Marcelo to pull from.

You can, of course, replace #7 and #8 with splitting it up into separate
patches and mailbombing if you have religious objections to BK.


But generally, we try to keep CVS working and stable. It's rare that it
is non-working for a prolonged period of time. You _should_ be able to
just drop it in to a recent 2.4 or 2.6 kernel and expect it to 'just
work'.

If I had hardware with interleaved AMD chips to hand, I'd have a look at
it (or at least wander round the corner to Arcom and take away David's
excuse :)

-- 
dwmw2

Re: kernel oops in cfi_cmdset_0002.c:do_write_one()...

[Steven Scholz]

David Vrabel wrote:
>> These error messages are from an ancient version of cfi_cmdset_0002.c.
>> They come from over-efficient software design that didn't consider the
>> quirks of how a flash chip works. Have you tried a recent version?
> 
> 
> Be aware that the more recent version from MTD CVS are broken for 
> interleaved chips.

What does "are broken" mean?
Just that autodetect of the flash type might not work? Or will read and 
write operations fail once the flash is detected correctly?

IOW: Could I use the MTD CVS with "interleaved chips" when I make sure my 
chips is correctly detected?

Thanks,

Steven

Re: kernel oops in cfi_cmdset_0002.c:do_write_one()...

[Jörn Engel]

On Fri, 9 April 2004 13:42:50 +0100, David Woodhouse wrote:
> On Thu, 2004-04-08 at 16:37 +0200, Jörn Engel wrote:
> > Don't blame him, this is the result of using CVS, which is designed
> > around the assumption of a single central repository.
> 
> Well, switching from CVS wouldn't necessarily make any difference. If I
> were using something like BitKeeper then there'd still only be one
> repository which I actually _care_ about, and the effect would be much
> the same. 

Right.  That I see the problem doesn't mean that I also see a
solution. :(

> The way the update goes is something like this...
> 
> 1. Wait till the MTD CVS code is actually stable. It's not at the
>    moment; cfi_cmdset_0002.c is known to be broken and I can't send it
>    off while it's like that.
> 
> 2. Look at the current version of the kernel I'm updating. For each file
>    in drivers/mtd and include/linux/mtd, take the version from CVS with
>    matching $Id$ tag -- the version from CVS which was last sent to
>    Linus or Marcelo for inclusion.
> 
> 3. Generate a patch between those versions in my CVS, and the version
>    currently in the mainstream kernel -- those are the patches which
>    were included in an upstream kernel but didn't go through me. Read
>    each change carefully, and see if it's applicable to _current_ CVS
>    code. If so, apply it. If not, discard it. If it's discarded because
>    the problem was already fixed, that's fine. If it's discarded because
>    it's just a bogus change, make a mental note of it; reversion of
>    upstream patches needs to be justified (or at least noted) in your
>    covering email because it's often done by accident when updating 
>    Linus' tree from an external CVS, and it's _very_ much frowned upon.
> 
> 4. Now you have all the upstream changes merged into current CVS. Drop
>    current CVS into a current Linux tree, take a diff between the two. 
>    Read it. All of it. Make cosmetic changes in CVS to reduce noise in
>    the patch you're planning to send.
> 
> 5. Read it again. 
> 
> 6. Build it. Test it if you're feeling conscientious.
> 
> 7. Read it again. Identify separate change sets from the whole thing;
>    'bk citool' is useful for this. Commit each set of changes separately
>    with an appropriate changelog. 
> 
> 8. Stick it in a BK tree somewhere for Linus or Marcelo to pull from.

2, 3 and 4 involve some manual work that could be automated, not sure
if you've already done that.  If you don't a human "patch assistant"
could help you, although I don't suspect many volunteers. ;)

7 is a bit philosophical.  It would be easier if patches were kept
seperate from the beginning, something CVS doesn't allow to do.  But
then you need to deal with dependences between them, and - worse -
make sure that seperate patches remain seperate.  It would make
merging easier, but the time before merging will be different, much
harder unless everyone has the discipline for it.

Propably this wouldn't work in your current mode of "everyone dump
into CVS", you'd have more work reading and refusing patches.  That
work is currently distributed, which makes your life easier somewhere
else.

Jörn

-- 
And spam is a useful source of entropy for /dev/random too!
-- Jasmine Strong

Re: kernel oops in cfi_cmdset_0002.c:do_write_one()...

[David Vrabel]

> 1. Wait till the MTD CVS code is actually stable. It's not at the
>    moment; cfi_cmdset_0002.c is known to be broken and I can't send it
>    off while it's like that.

Revert cfi_cmdset_0002.c back to 1.94?  1.95 is the change that appears 
to have broken interleaved chips.

David

Re: kernel oops in cfi_cmdset_0002.c:do_write_one()...

[David Woodhouse]

On Tue, 2004-04-13 at 11:31 +0200, Jörn Engel wrote:
> > Well, switching from CVS wouldn't necessarily make any difference. If I
> > were using something like BitKeeper then there'd still only be one
> > repository which I actually _care_ about, and the effect would be much
> > the same. 
> 
> Right.  That I see the problem doesn't mean that I also see a
> solution. :(

The point is that there isn't a 'solution', and that the problem isn't
CVS.

The 'problem' is the task at hand -- maintaining a kernel subsystem that
only a few people care deeply about.

I think it needs to be done outside Linus' tree, for the following
reasons:
 - Patches are of varying quality and I don't have time to vet or 
   rewrite them all before timely inclusion. 
 - People don't use the current kernels, even though they should.
 - Having a development branch which gets fairly widespead testing
   is useful.

Letting people commit directly to CVS seems to work fairly well. It gets
stuff tested without me having to do everything myself, and makes it
nice and easy for people to take over 'maintenance' of individual
drivers. I think we manage to keep a fairly reasonable balance of
stability in CVS too -- it's not often broken.

The downside is that it does take a while for Linus' tree to get
updated, and that's partly because the procedure involves real human
intervention to vet all the changes that have gone in.

But that vetting has to be done _whatever_ happens. The question is do
we restrict the number of people who can commit to CVS, and make people
submit patches for vetting before they're committed, or do we let them
commit first and then improve it over time if such is necessary?

Again, I think we have the balance right. People commit reasonable
patches to CVS, and occasionally I may revert them if I _really_ object.
But they don't have to go through me or other people first, and there's
not a bottleneck during _submission_.

I'm always open to suggestions for how to improve the process -- but I
don't have any ideas. If someone's offended that a particular feature
hasn't made it into Linus' tree, the best course of action I can
recommend is to go through my update procedure themselves, merging
changes from Linus' tree into CVS in a way which works in both 2.4 and
2.6, and then producing a _clean_ patch which Linus will accept, and
which updates his tree to match what's now in CVS. Then mail it to me
and I'll look at it. If it's all fine, all I need to do is look, and I
can say "OK, Linus please apply this patch". 
> 2, 3 and 4 involve some manual work that could be automated, not sure
> if you've already done that.  If you don't a human "patch assistant"
> could help you, although I don't suspect many volunteers. ;)

It's not a lot of manual work. Lots of 'grep $Id: | cut ... xargs cvs'
but that's not really 'manual'.

> 7 is a bit philosophical.  It would be easier if patches were kept
> seperate from the beginning, something CVS doesn't allow to do.  But
> then you need to deal with dependences between them, and - worse -
> make sure that seperate patches remain seperate.  It would make
> merging easier, but the time before merging will be different, much
> harder unless everyone has the discipline for it.

It's not so hard. I rarely want to make changes to any given file in
more than one changeset. Before BK I could just make a single big
'update' patch then open it in emacs and cut'n'paste into separate
files. With BK I use citool to commit selected files in different csets.

> Propably this wouldn't work in your current mode of "everyone dump
> into CVS", you'd have more work reading and refusing patches.  That
> work is currently distributed, which makes your life easier somewhere
> else.

Indeed -- and it makes other peoples lives easier too because they can
actually move on without waiting for me (or other trustworthy people to
whom I could delegate the task -- they exist, but they're probably not
blessed with any more copious quantities of free time than I am).

-- 
dwmw2