tricky challenge for getting round level-driven interrupt problem: help!

tricky challenge for getting round level-driven interrupt problem: help!

[Luke Kenneth Casson Leighton]

hi, please kindly respond cc to me because i am subscribed to the
lists for post-only-and-view-archives-on-demand-to-minimise-overload
purposes.

i have a particularly acute and knotty computing problem involving
a stupid hardware design fault in a cirrus logic "maverick" EDB 7134
ARM processor (max 90Mhz).

it only does level-based interrupts and i need to create a driver
that does two-way 8-bit data communication.

i would genuinely appreciate some advice from people with
more experience than i on how to go about getting round
this stupid hardware design - in order to make a RELIABLE,
non-race-conditioned kernel driver.


hardware
--------

connected to the ARM's 8-bit port is a 6Mhz p16f877a PIC processor.

connected to the serial and other ports of the PIC processor are
various peripherals, including an LCD, a GPS satellite receiver,
an accelerometer and the battery level / charger detector.

on the PIC is some assembler code that:
--------------------------------------

* merges GPS, Accelerometer, battery and charger information
  into a data stream that goes out the 8-bit port of the PIC and
  in on the ARM's 8-bit port.

* receives instructions from the ARM down the same 8-bit port,
  in a custom-designed length-encoded data stream that tells
  the PIC what to put on the LCD, and where.

interrupts are generated as follows:
------------------------------------

* on the PIC, a "read" interrupt can be asserted to the ARM.
  it's xxxxing level-based.

* on the PIC, a "write" interrupt can be asserted to the ARM.
  it's xxxxing level-based.

* on the ARM, a SINGLE interrupt can be generated to the PIC
  by EITHER reading OR writing to the PIC's 8-bit port.
  
  unlike the stupid xxxxing ARM, it's edge triggered (thank god).

interrupts are cleared as follows:
---------------------------------

* on the PIC, in the [single!] ISR routine, if the PIC knows
  that it was doing a read, it resets the "read" interrupt
  flag to the ARM.

  [the problem is obvious: a level-based interrupt could fail
   to be acknowledged, could be masked out and accidentally
   regenerated, and we are into "nightmare" scenario time]

* on the PIC, in the [single!] ISR routine, if the PIC knows
  that it was doing a read, it resets the "read" interrupt
  flag to the ARM.

  [the problem is obvious: a level-based interrupt could fail
   to be acknowledged, could be masked out and accidentally
   regenerated, and we are into "nightmare" scenario time]

* on the ARM, in the READ isr routine, the PIC "reads" the
  byte, THIS ACT generates an interrupt to the PIC, which
  the ARM then acknowledges by waiting - in a tight loop - 
  for the PIC to clear the "read" interrupt flag.

  [this is the old code btw, not the new code: for brevity
   i have not described exactly how bad the code is]

* on the ARM, in the WRITE isr routine, similar situation
  as for "read".


the protocol that i designed to overcome the race condition
nightmare is as follows:

* ARM and PIC ****MUST**** exchange read and write bytes, interleaved.

* if the ARM does not have anything to write [to the LCD] at the
  time that a read is to be carried out, it sends a "dummy"
  or "null" encoded data stream indicating to the PIC that it
  is receiving data of zero length.

  the sole purpose of initiating this "i-am-sending-you-zero-bytes"
  dummy stream of bytes is to keep the read-write-read-write....
  cycle going.

* if the PIC does not have anything to be read [from the GPS
  and other peripherals] then it sends "0xff" instead.  the ARM
  receives this non-ascii byte and knows that it must throw it
  away.

  the sole purpose of sending this "non-ascii" byte is to keep
  the read-write-read-write... cycle going.

* if there is both read and write data to be exchanged, everything
  is hunky-dory.

* if there is no data to be exchanged, everything is hunky-dory.


here's where i have got to, and where i am stuck:
----------------------------------------------------

on the ARM, i have cut/paste the code from sonypi.c to create
a poll_wait "read" driver.  it is pretty much exactly the same
structure as sonypi.c, as far as "read" is concerned.

so, when reading, I/O is in non-blocking mode, the queue is
empty, a WAITQUEUE is woken up, present kernel process is made
"TASK_INTERRUPTIBLE", schedule()d, a read interrupt is [eventually]
generated and read data added to queue, then pic_misc_poll() detects
data now present, and "WAKES UP" the waiting blocking process.

everything hunky-dory.


here's where i have got to, and where i am stuck:
----------------------------------------------------

i don't know how to "wake up" an equivalent write process.

because there isn't one.

the situation where there is simultaneous blocking-read and
blocking-write _could_ be covered by duplicating the sonypi.c code
_again_ for writing.

that leaves the situations where there is ONLY read occurring.

bearing in mind that reads must be interleaved with writes,
and those reads must be intitiated from inside the "read"
interrupt service routine.... what the hell do i do?

i think i have a clue in that the "old" pic_write() code would
unmask the Write Interrupt, and a "write" interrupt would
immediately occur - then once we'd done, Write Interrupts
would be re-masked again.


any advice really appreciated because this is one mean comp-sci
classic that really only _actually_ occurs in real life when
there's a fuckup in the hardware design and you don't have
any choice but to make it work :)

cheers,

l.

-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--

Re: tricky challenge for getting round level-driven interrupt problem: help!

[Alan Cox]

On Maw, 2005-05-03 at 22:56, Luke Kenneth Casson Leighton wrote:
> it only does level-based interrupts and i need to create a driver
> that does two-way 8-bit data communication.

Level triggered is generally considered a feature by people less than
200 years old 8)

> i would genuinely appreciate some advice from people with
> more experience than i on how to go about getting round
> this stupid hardware design - in order to make a RELIABLE,
> non-race-conditioned kernel driver.

Assuming you are using the core Linux IRQ code then you shouldn't have
any great problem. The basic rules with level triggered IRQ are that 

- You must ack the IRQ on the device to clear it either specifically or
as part of some other process.
- The device must raise the IRQ again if the irq condition is present
*at* or after the point of the IRQ ack (or in many implementations 'just
before' in fact)

the core Linux code deals with masking on the controller to ensure IRQ's
dont get called recursively, and ensuring an IRQ gets rechecked/recalled
on the unmask/exit path if it is still asserted.

So an implementation is much like a serial port fifo, assert the IRQ
until the PC has fetched all the bits. In fact the IRQ line is
essentially "NOT(fifo_empty)" and rechecked each time a byte is fetched.

*WAY* simpler than nasty edge triggered stuff 8)

Alan

Re: tricky challenge for getting round level-driven interrupt problem: help!

[Luke Kenneth Casson Leighton]

alan, thanks for responding directly.

the internet is down at the moment (at least, bt's bit is) so
your continued direct responses, if you have the time, greatly
appreciated.

i can't get http or ssh access to _anywhere_, but bizarrely,
smtp is getting through.

[sod bt.  royally.]

On Wed, May 04, 2005 at 02:50:01AM +0100, Alan Cox wrote:

> On Maw, 2005-05-03 at 22:56, Luke Kenneth Casson Leighton wrote:
> > it only does level-based interrupts and i need to create a driver
> > that does two-way 8-bit data communication.
> 
> Level triggered is generally considered a feature by people less than
> 200 years old 8)
 
 lots of experience, then, you'd say?  my experience with doing this
 kind of stuff is kinda limited (doesn't stop me trying) so your
 response is really appreciated.

> > i would genuinely appreciate some advice from people with
> > more experience than i on how to go about getting round
> > this stupid hardware design - in order to make a RELIABLE,
> > non-race-conditioned kernel driver.
> 
> Assuming you are using the core Linux IRQ code 

 yep.

> then you shouldn't have
> any great problem. The basic rules with level triggered IRQ are that 
> 
> - You must ack the IRQ on the device to clear it either specifically or
> as part of some other process.
> - The device must raise the IRQ again if the irq condition is present
> *at* or after the point of the IRQ ack (or in many implementations 'just
> before' in fact)
> 
> the core Linux code deals with masking on the controller to ensure IRQ's
> dont get called recursively, and ensuring an IRQ gets rechecked/recalled
> on the unmask/exit path if it is still asserted.
> 
> So an implementation is much like a serial port fifo, assert the IRQ
> until the PC has fetched all the bits. In fact the IRQ line is
> essentially "NOT(fifo_empty)" and rechecked each time a byte is fetched.
 
 okay.

> *WAY* simpler than nasty edge triggered stuff 8)
 
 bleh :)

 okay.

 i believe i get it: you raise a level-triggered interrupt which _stays_
 raised until such time as your fifo is empty.

 the original protocol that was written [by somebody even less
 experienced than i] was designed the way it is because the PIC chip
 that's connected to the ARM only has *one* interrupt.

 that interrupt is triggered:

 * on a port read (there are four 8-bit ports!)
 * on a port write 
 * by the serial port
 * by the timer

 [skip this bit if you're busy, it's just background info]

	 so there's now a _thousand_ lines of hand-written assembly code
	 which took well over a year to write and debug on the PIC, that
	 implements the state-machine to deal with the GPS on the serial
	 port, interleaving the "device status" info into the GPS data
	 stream, getting the GPS data stream over the 8-bit port and
	 receiving LCD commands over the 8-bit port, interpreting them
	 and then feeding the LCD.

 okay.

 so.

 assuming that i can handle the pic having only a single interrupt, then
 does the process go something like this (for read, e.g.):

 1) the ARM unmasks read interrupts when it's ready.

 2) when the PIC has data to be read (which is only ever a single byte),
   it asserts the level-triggered read interrupt to the ARM.

 3) the ARM receives the interrupt, and reads the byte.  the _act_ of
   reading the byte causes an edge-triggered interrupt to the PIC.


 5) the first thing the PIC does in its [one] ISR is to de-assert the
   read interrupt.  why? because there is only a one byte buffer.

 
 we have an additional step in the chain - step 4 - which is:

 4) in the ISR, the ARM goes into a tight loop checking that the 
    PIC has cleared the level-triggered interrupt.
 

 the reason for checking this is so that when we come out of the
 ISR on the ARM, the PIC is *SOOO* slow - it's only 6mhz after
 all - that sometimes (frequently, in fact - about 1 in every
 50 times) it hasn't got round to clearing the level-driven
 interrupt by the time we come out of the ARM ISR (!)

 the problem we're having is actually worse than that.

 sometimes something goes screwy and _despite_ having this loop-check,
 we get duplicated characters.

 sometimes, even worse, we lose sync, and the ARM sits there in the
 loop...


 i was getting _so_ fed up with this that i was thinking that there
 was a problem with the approach being taken.

 hence the redesign to do alternate read-write-read-write, and making
 reads exclusive of writes, etc.

 awful.  just awful :)


 ... so - in your opinion, alan, is the old approach we had
 actually _on_ the right lines?

 also, are you going to ukuug in august, it being _in_
 aberystwyth and all :)

 l.

Re: tricky challenge for getting round level-driven interrupt problem: help!

[Alan Cox]

On Mer, 2005-05-04 at 21:58, Luke Kenneth Casson Leighton wrote:
>  i believe i get it: you raise a level-triggered interrupt which _stays_
>  raised until such time as your fifo is empty.

Bingo. It only goes away when the chip really has nothing left to say.

>  all - that sometimes (frequently, in fact - about 1 in every
>  50 times) it hasn't got round to clearing the level-driven
>  interrupt by the time we come out of the ARM ISR (!)

So you'll poll again and find there is no pending work to do.

>  hence the redesign to do alternate read-write-read-write, and making
>  reads exclusive of writes, etc.

and maybe even turn the IRQ off and use a timer if its slow and not
sensitive to latency.. ?

>  ... so - in your opinion, alan, is the old approach we had
>  actually _on_ the right lines?

level triggered IRQ does sort of expect the other end responds promptly
to be efficient as opposed to merely reliable.

>  also, are you going to ukuug in august, it being _in_
>  aberystwyth and all :)

Its not in Aberystwyth, but I might be. Its in Swansea 8)

Re: tricky challenge for getting round level-driven interrupt problem: help!

[Luke Kenneth Casson Leighton]

On Wed, May 04, 2005 at 10:43:35PM +0100, Alan Cox wrote:
> On Mer, 2005-05-04 at 21:58, Luke Kenneth Casson Leighton wrote:
> >  i believe i get it: you raise a level-triggered interrupt which _stays_
> >  raised until such time as your fifo is empty.
> 
> Bingo. It only goes away when the chip really has nothing left to say.
> 
> >  all - that sometimes (frequently, in fact - about 1 in every
> >  50 times) it hasn't got round to clearing the level-driven
> >  interrupt by the time we come out of the ARM ISR (!)
> 
> So you'll poll again and find there is no pending work to do.

 oh.

 ah.

 wait - this might be the bit i don't get 

 when you say poll again, do you mean poll again inside the ISR?

 so:
 
 * we do the read (which creates the interrupt to the PIC) and
   then sit there polling the level-driven interrupt status
   register
 
 * the PIC, having no more bytes to write, clears the ARM
   level-driven interrupt.

 * the ARM detects the change of the level-driven interrupt
   status register

 * the ARM pauses for, oh, i dunno... mmm... 6mhz equals 166ns
   so we call udelay(10) or something ridiculously long...

 * the ARM then checks the level-driven interrupt status register
   AGAIN, and if it's clear, goes "oh, we ain't gonna get no
   more data".

 something like that? [am trying it now, anyway - on the basis that it
 can't hurt :)]

 i'm not being deliberately thick - honest - i've just never had to
 do this sort of thing before.

 btw alan where would you recommend i read up on this type of thing?
 [_please_ don't say "on the inside of my skull" i'll be tempted to
 go get a saw and a spoon i've _so_ got to get this to work...]
 

> >  hence the redesign to do alternate read-write-read-write, and making
> >  reads exclusive of writes, etc.
> 
> and maybe even turn the IRQ off and use a timer if its slow and not
> sensitive to latency.. ?
 
 tried that last month [i'm not sending to lkml about this as a first
 resort - honest!]
 
 the baud rate from the GPS is 4800 baud, so that's 600 chars/sec.

 the jiffies timer on the ARM is ... er... 250? per second?

 so ... hey, yeh, that would explain why i saw about 1
 character in 3 :)



> >  ... so - in your opinion, alan, is the old approach we had
> >  actually _on_ the right lines?
> 
> level triggered IRQ does sort of expect the other end responds promptly
> to be efficient as opposed to merely reliable.

 ... and a 6mhz processor vs a 90mhz processor... 

> >  also, are you going to ukuug in august, it being _in_
> >  aberystwyth and all :)
> 
> Its not in Aberystwyth, but I might be. Its in Swansea 8)

 ahhh :)

 glad i checked then, 'cos up until 30 seconds ago i was gonna
 drive to aber.

 *wonders why the hell i've been let loose on this project...*

Re: tricky challenge for getting round level-driven interrupt problem: help!

[Luke Kenneth Casson Leighton]

On Wed, May 04, 2005 at 10:43:35PM +0100, Alan Cox wrote:

> >  hence the redesign to do alternate read-write-read-write, and making
> >  reads exclusive of writes, etc.
> 
> and maybe even turn the IRQ off and use a timer if its slow and not
> sensitive to latency.. ?

 good suggestion...
 
 been there, tried that [i really _am_ sending to lkml as
 last resort, not first!]

 jiffies equals approx 250? per second?

 baud rate from PIC is determined by GPS - 4800 baud - approx
 600 per second.

 so that'd explain why i only got one character every 3.

 *cold sweat*.  

 i could always use the FIQ, which will be running off the back
 of the Audio DAC/ADC's interrupts, 8khz....

 *shudder*...

Re: tricky challenge for getting round level-driven interrupt problem: help!

[Luke Kenneth Casson Leighton]

On Wed, May 04, 2005 at 10:43:35PM +0100, Alan Cox wrote:

> On Mer, 2005-05-04 at 21:58, Luke Kenneth Casson Leighton wrote:
> >  i believe i get it: you raise a level-triggered interrupt which _stays_
> >  raised until such time as your fifo is empty.
> 
> Bingo. It only goes away when the chip really has nothing left to say.
> 
> >  all - that sometimes (frequently, in fact - about 1 in every
> >  50 times) it hasn't got round to clearing the level-driven
> >  interrupt by the time we come out of the ARM ISR (!)
> 
> So you'll poll again and find there is no pending work to do.
> 
> >  hence the redesign to do alternate read-write-read-write, and making
> >  reads exclusive of writes, etc.
> 
> and maybe even turn the IRQ off and use a timer if its slow and not
> sensitive to latency.. ?

 okay.

 you will be pleased to know that the switch-off-irqs-in-the-isr
 and then switch-it-back-on-in-the-FIQ approach works.

 *gibber*

 basically i am doing this:

 * in the ARM interrupt read routine, first thing i do is mask
   read interrupts.

 * i then set up the ARM port for "read", including disabling the
   keyboard which shares some of the port lines.

 * i then read the byte (which causes an edge-triggered interrupt
   to the PIC, which then sets about clearing the interrupt).
 
 * i then re-enable the keyboard.

 * at the end of the routine, i then set a flag.


 some time later, within 1/8000th of a second, this flag is checked.
 if it's set, read interrupts are unmasked.

 
 ... btw when you say "using the linux interrupt management" routines,
 what exactly do you mean?

 at the moment (because that's what was done when i stepped in
 to sort out this code) i am directly accessing the INTMR1 and
 INTSR1 registers - _not_ calling enable_irq() or disable_irq()
 or other)

 (and in the FIQ handler of course i _can't_ call those functions).

 is that a problem?

 l.

Re: tricky challenge for getting round level-driven interrupt problem: help!

[Luke Kenneth Casson Leighton]

hi: just a fyi.

i do not know, cannot tell, and do not care [ItNowWorks(tm)] why what i
have done manages to GetItToWork(tm) - i am merely providing you with
information such that _other people_ may dissect this issue, may look at
this in six months/years and go "oh yeah, maybe i should try that too".

in the write communications, i have REMOVED the use of interrupts and am
PURELY using the INTMR1 bit for polling purposes (!!!)

the code goes something like this:

int pic_misc_write( ..... char *data, int len)
{
	for (i = 0; i < data; i++)
	{
		pic_queue_byte(data[i]);
		pic_writable_int(0, 0, NULL);
	}
}

ha ha, i hear you say, very funny.

well, in pic_writable_int - which USED to be called as an interrupt
handler, i have this:

irq_return_t pic_writable_int(int irq, ....)
{
	/* check that the PIC is ready for us to proceed */
	while (!(clps_readl(INTMR) & 0x40)) {
	}
	...
	...

	/* place the byte on the 
	clps_writeb(pic_get_queued_byte(), PORT_A_DATA_REGISTER);

	/* this generates an interrupt to the PIC */
	clps_writeb(0x...., PORT_A_SOMETHING_REGISTER);

	/* ... which we sit there twiddling our thumbs waiting for
	   it to be cleared 
	 */
	while ((clps_readl(INTMR) & 0x40)) {
	}
}

ironically, it's _still_ not 100% reliable: about 1 in every 400 bytes
gets corrupted.

but that's probably due to bugs in the PIC assembly code, which i can
live with / get these guys to fix, or it's to do with the setup timing
of the port directions - again, i can deal with that.

... the question is - what the _hell_ is going on with the linux 2.6
level-based interrupt handling / my-braindead-way-of-using-them
that causes the above approach to be [virtually] 100% reliable
as compared to a 95%-reliable cock-up?

l.

On Wed, May 04, 2005 at 09:58:31PM +0100, Luke Kenneth Casson Leighton wrote:
> alan, thanks for responding directly.
> 
> the internet is down at the moment (at least, bt's bit is) so
> your continued direct responses, if you have the time, greatly
> appreciated.
> 
> i can't get http or ssh access to _anywhere_, but bizarrely,
> smtp is getting through.
> 
> [sod bt.  royally.]
> 
> On Wed, May 04, 2005 at 02:50:01AM +0100, Alan Cox wrote:
> 
> > On Maw, 2005-05-03 at 22:56, Luke Kenneth Casson Leighton wrote:
> > > it only does level-based interrupts and i need to create a driver
> > > that does two-way 8-bit data communication.
> > 
> > Level triggered is generally considered a feature by people less than
> > 200 years old 8)
>  
>  lots of experience, then, you'd say?  my experience with doing this
>  kind of stuff is kinda limited (doesn't stop me trying) so your
>  response is really appreciated.
> 
> > > i would genuinely appreciate some advice from people with
> > > more experience than i on how to go about getting round
> > > this stupid hardware design - in order to make a RELIABLE,
> > > non-race-conditioned kernel driver.
> > 
> > Assuming you are using the core Linux IRQ code 
> 
>  yep.
> 
> > then you shouldn't have
> > any great problem. The basic rules with level triggered IRQ are that 
> > 
> > - You must ack the IRQ on the device to clear it either specifically or
> > as part of some other process.
> > - The device must raise the IRQ again if the irq condition is present
> > *at* or after the point of the IRQ ack (or in many implementations 'just
> > before' in fact)
> > 
> > the core Linux code deals with masking on the controller to ensure IRQ's
> > dont get called recursively, and ensuring an IRQ gets rechecked/recalled
> > on the unmask/exit path if it is still asserted.
> > 
> > So an implementation is much like a serial port fifo, assert the IRQ
> > until the PC has fetched all the bits. In fact the IRQ line is
> > essentially "NOT(fifo_empty)" and rechecked each time a byte is fetched.
>  
>  okay.
> 
> > *WAY* simpler than nasty edge triggered stuff 8)
>  
>  bleh :)
> 
>  okay.
> 
>  i believe i get it: you raise a level-triggered interrupt which _stays_
>  raised until such time as your fifo is empty.
> 
>  the original protocol that was written [by somebody even less
>  experienced than i] was designed the way it is because the PIC chip
>  that's connected to the ARM only has *one* interrupt.
> 
>  that interrupt is triggered:
> 
>  * on a port read (there are four 8-bit ports!)
>  * on a port write 
>  * by the serial port
>  * by the timer
> 
>  [skip this bit if you're busy, it's just background info]
> 
> 	 so there's now a _thousand_ lines of hand-written assembly code
> 	 which took well over a year to write and debug on the PIC, that
> 	 implements the state-machine to deal with the GPS on the serial
> 	 port, interleaving the "device status" info into the GPS data
> 	 stream, getting the GPS data stream over the 8-bit port and
> 	 receiving LCD commands over the 8-bit port, interpreting them
> 	 and then feeding the LCD.
> 
>  okay.
> 
>  so.
> 
>  assuming that i can handle the pic having only a single interrupt, then
>  does the process go something like this (for read, e.g.):
> 
>  1) the ARM unmasks read interrupts when it's ready.
> 
>  2) when the PIC has data to be read (which is only ever a single byte),
>    it asserts the level-triggered read interrupt to the ARM.
> 
>  3) the ARM receives the interrupt, and reads the byte.  the _act_ of
>    reading the byte causes an edge-triggered interrupt to the PIC.
> 
> 
>  5) the first thing the PIC does in its [one] ISR is to de-assert the
>    read interrupt.  why? because there is only a one byte buffer.
> 
>  
>  we have an additional step in the chain - step 4 - which is:
> 
>  4) in the ISR, the ARM goes into a tight loop checking that the 
>     PIC has cleared the level-triggered interrupt.
>  
> 
>  the reason for checking this is so that when we come out of the
>  ISR on the ARM, the PIC is *SOOO* slow - it's only 6mhz after
>  all - that sometimes (frequently, in fact - about 1 in every
>  50 times) it hasn't got round to clearing the level-driven
>  interrupt by the time we come out of the ARM ISR (!)
> 
>  the problem we're having is actually worse than that.
> 
>  sometimes something goes screwy and _despite_ having this loop-check,
>  we get duplicated characters.
> 
>  sometimes, even worse, we lose sync, and the ARM sits there in the
>  loop...
> 
> 
>  i was getting _so_ fed up with this that i was thinking that there
>  was a problem with the approach being taken.
> 
>  hence the redesign to do alternate read-write-read-write, and making
>  reads exclusive of writes, etc.
> 
>  awful.  just awful :)
> 
> 
>  ... so - in your opinion, alan, is the old approach we had
>  actually _on_ the right lines?
> 
>  also, are you going to ukuug in august, it being _in_
>  aberystwyth and all :)
> 
>  l.
> 

-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--