Problems with latency

Problems with latency

[Zik Saleeba]

[-- Attachment #1.1: Type: text/plain, Size: 1399 bytes --]

I'm writing a driver for the NXP sc16is752 serial chip. This chip is
accessed via SPI. I've actually had a working driver for quite some time now
but I've been having problems with it since I changed from 2.6.16 to
2.6.24.Occasionally the driver fails due to its 64 byte receive buffer
overflowing
before I can get the data.

I've set the serial chip's receive buffer interrupt level to 16 bytes - so
after I've received 16 bytes I have to get the data before the buffer
overflows its 64 byte limit. This isn't too demanding - I need to have it
done within 8ms of getting the interrupt at 57600bps. The tricky part is
that I need to do three separate register reads to get status before I can
do the final buffer read so it's actually four SPI transactions. And what's
more the chip has two serial ports so I need to check both - which makes it
up to eight SPI transactions.

Normally this is absolutely no problem - the bottom half work queue fires up
and the SPI subsystem on my pxa2xx romps it in. Once in a while however the
latency between the interrupt and removing the data is much, much longer and
the chip's receive buffer overflows. This happens enough that it makes the
driver unusable at higher bit rates.

Does anyone have any suggestions what might be causing these random latency
jumps, and also whether it might be related to work queue latency or to
pxa2xx spi delays?

Thanks,
Zik

[-- Attachment #1.2: Type: text/html, Size: 1503 bytes --]

[-- Attachment #2: Type: text/plain, Size: 247 bytes --]

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php

[-- Attachment #3: Type: text/plain, Size: 210 bytes --]

_______________________________________________
spi-devel-general mailing list
spi-devel-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/spi-devel-general

Re: Problems with latency

[Ned Forrester]

Zik Saleeba wrote:
> I'm writing a driver for the NXP sc16is752 serial chip. This chip is
> accessed via SPI. I've actually had a working driver for quite some time now
> but I've been having problems with it since I changed from 2.6.16 to
> 2.6.24.Occasionally the driver fails due to its 64 byte receive buffer
> overflowing
> before I can get the data.
> 
> I've set the serial chip's receive buffer interrupt level to 16 bytes - so
> after I've received 16 bytes I have to get the data before the buffer
> overflows its 64 byte limit. This isn't too demanding - I need to have it
> done within 8ms of getting the interrupt at 57600bps. The tricky part is
> that I need to do three separate register reads to get status before I can
> do the final buffer read so it's actually four SPI transactions. And what's
> more the chip has two serial ports so I need to check both - which makes it
> up to eight SPI transactions.
> 
> Normally this is absolutely no problem - the bottom half work queue fires up
> and the SPI subsystem on my pxa2xx romps it in. Once in a while however the
> latency between the interrupt and removing the data is much, much longer and
> the chip's receive buffer overflows. This happens enough that it makes the
> driver unusable at higher bit rates.
> 
> Does anyone have any suggestions what might be causing these random latency
> jumps, and also whether it might be related to work queue latency or to
> pxa2xx spi delays?

Perhaps you have found a solution by now.  I have been away.

The only suggestion I have is a general one, not specifically related to
the driver.  In my application, I read from an SPI device at 11MBit/sec,
and write the data to an NFS server elsewhere on the network.  I found
that if I did not do frequent fdatasync() calls on the NFS file from my
user-space code, then Linux would simply queue huge amounts of data in
the PXA255's 64MB RAM, and would not actually write to the network until
the RAM was nearly full.  When it did finally write to the network,
there were huge latencies (nearly a second) in interrupt service to the
SPI.  By forcing it to write in smaller chunks, the latency problem
disappeared.  I have no idea whether this could be related to your problem.

-- 
Ned Forrester                                       nforrester-/d+BM93fTQY@public.gmane.org
Oceanographic Systems Lab                                  508-289-2226
Applied Ocean Physics and Engineering Dept.
Woods Hole Oceanographic Institution          Woods Hole, MA 02543, USA
http://www.whoi.edu/sbl/liteSite.do?litesiteid=7212
http://www.whoi.edu/hpb/Site.do?id=1532
http://www.whoi.edu/page.do?pid=10079


-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08

Re: Problems with latency

[Zik Saleeba]

[-- Attachment #1.1: Type: text/plain, Size: 3948 bytes --]

Thanks Ned, I finally found the solution to my problem. It turned out that
the problem was in fact nothing to do with the receive side of the system
where the FIFO overflows were occurring. The problem was that the standard
linux tty driver was very aggressively calling a transmit flush on every
single character sent. This works well when your serial chip is connected
directly on the processor bus but turns out to be quite a bad thing when
your serial chip is connected via SPI.

The very large number of SPI transmits was causing a huge amount of
interrupt activity, which was in turn occasionally causing delays in getting
to the receive side work queue. These delays were long enough that the
receive FIFO would overflow and it was all bad from there.

I'm not sure what changed between 2.6.16 and 2.6.24 which made this become a
critical problem - quite possibly something subtle could have had this
side-effect. I solved the problem by aggregating the transmit characters and
sending them in larger groups. This seems to have fixed the problem.

Thanks for your follow-up,
Zik

On Thu, Jul 10, 2008 at 12:22 AM, Ned Forrester <nforrester-/d+BM93fTQY@public.gmane.org> wrote:

> Zik Saleeba wrote:
> > I'm writing a driver for the NXP sc16is752 serial chip. This chip is
> > accessed via SPI. I've actually had a working driver for quite some time
> now
> > but I've been having problems with it since I changed from 2.6.16 to
> > 2.6.24.Occasionally the driver fails due to its 64 byte receive buffer
> > overflowing
> > before I can get the data.
> >
> > I've set the serial chip's receive buffer interrupt level to 16 bytes -
> so
> > after I've received 16 bytes I have to get the data before the buffer
> > overflows its 64 byte limit. This isn't too demanding - I need to have it
> > done within 8ms of getting the interrupt at 57600bps. The tricky part is
> > that I need to do three separate register reads to get status before I
> can
> > do the final buffer read so it's actually four SPI transactions. And
> what's
> > more the chip has two serial ports so I need to check both - which makes
> it
> > up to eight SPI transactions.
> >
> > Normally this is absolutely no problem - the bottom half work queue fires
> up
> > and the SPI subsystem on my pxa2xx romps it in. Once in a while however
> the
> > latency between the interrupt and removing the data is much, much longer
> and
> > the chip's receive buffer overflows. This happens enough that it makes
> the
> > driver unusable at higher bit rates.
> >
> > Does anyone have any suggestions what might be causing these random
> latency
> > jumps, and also whether it might be related to work queue latency or to
> > pxa2xx spi delays?
>
> Perhaps you have found a solution by now.  I have been away.
>
> The only suggestion I have is a general one, not specifically related to
> the driver.  In my application, I read from an SPI device at 11MBit/sec,
> and write the data to an NFS server elsewhere on the network.  I found
> that if I did not do frequent fdatasync() calls on the NFS file from my
> user-space code, then Linux would simply queue huge amounts of data in
> the PXA255's 64MB RAM, and would not actually write to the network until
> the RAM was nearly full.  When it did finally write to the network,
> there were huge latencies (nearly a second) in interrupt service to the
> SPI.  By forcing it to write in smaller chunks, the latency problem
> disappeared.  I have no idea whether this could be related to your problem.
>
> --
> Ned Forrester                                       nforrester-/d+BM93fTQY@public.gmane.org
> Oceanographic Systems Lab                                  508-289-2226
> Applied Ocean Physics and Engineering Dept.
> Woods Hole Oceanographic Institution          Woods Hole, MA 02543, USA
> http://www.whoi.edu/sbl/liteSite.do?litesiteid=7212
> http://www.whoi.edu/hpb/Site.do?id=1532
> http://www.whoi.edu/page.do?pid=10079
>
>

[-- Attachment #1.2: Type: text/html, Size: 5022 bytes --]

[-- Attachment #2: Type: text/plain, Size: 347 bytes --]

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08

[-- Attachment #3: Type: text/plain, Size: 210 bytes --]

_______________________________________________
spi-devel-general mailing list
spi-devel-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/spi-devel-general