pmac_zilog debugging ...

pmac_zilog debugging ...

[Kevin Diggs]

Hi,

	If I use a command similar to:

pppd ttyS0 1200 satellites: netmask 255.255.255.0 lock crtscts mru 1064 noauth debug kdebug 7 
logfile /tmp/pppd.log local

to connect an 8600 to a laptop via ppp the link will lock up in short
order from "payloaded" pings. Any advice on how to figure out where it
is locking up? This command works fine to connect two x86 laptops. At
1200 it does take a while for an xterm to show up, though.

kevin

Re: pmac_zilog debugging ...

[Benjamin Herrenschmidt]

On Fri, 2008-11-07 at 13:38 -0800, Kevin Diggs wrote:
> to connect an 8600 to a laptop via ppp the link will lock up in short
> order from "payloaded" pings. Any advice on how to figure out where it
> is locking up? This command works fine to connect two x86 laptops. At
> 1200 it does take a while for an xterm to show up, though.

Flow control might be busted. You may need to revert the polarity of the
CTS line sampling. Let me know what you find out.

Cheers,
Ben.

Re: pmac_zilog debugging ...

[Kevin Diggs]

Benjamin Herrenschmidt wrote:
> On Fri, 2008-11-07 at 13:38 -0800, Kevin Diggs wrote:
> 
>>to connect an 8600 to a laptop via ppp the link will lock up in short
>>order from "payloaded" pings. Any advice on how to figure out where it
>>is locking up? This command works fine to connect two x86 laptops. At
>>1200 it does take a while for an xterm to show up, though.
> 
> 
> Flow control might be busted. You may need to revert the polarity of the
> CTS line sampling. Let me know what you find out.
> 
> Cheers,
> Ben.
> 
I put the YDL 2.1 disk with it's 2.4.31 kernel back in the beast. With the
macserial driver ppp works flawlessly. I can connect to my ISP using one of
those Global Village modems. I can also torture a Nec VersaM100 laptop at
115200.

Using 2.6.27 and pmac_zilog I can't even run:

	xdpyinfo -display pm8600bs:0

after telnetting into afore mentioned abused laptop.

There are some debug macros that are in pmac_zilog. I have also added a few
new ones in an attempt to see where it is locking up. The following lines
are the result of the DEBUG_HARD macro (interrupt debug):

pmac_zilog 0.00013000:ch-b: irq, r3: 2
pmac_zilog 0.00013000:ch-b: irq done.
pmac_zilog 0.00013000:ch-b: irq, r3: 4
pmac_zilog 0.00013000:ch-b: pmz_receive_chars() -> 1 (loops 1)
pmac_zilog 0.00013000:ch-b: irq done.
pmac_zilog 0.00013000:ch-b: irq, r3: 0
pmac_zilog 0.00013000:ch-b: irq done.
pmac_zilog 0.00013000:ch-b: irq, r3: 2
pmac_zilog 0.00013000:ch-b: irq done.

I think r3 is the interrupt status register. Note the '0' for the next to
last interrupt. I think this is a pattern: a 0 (???), a 2 (tx), then
locked.

Here are some metrics.

2.4.31 macserial @ 115,200:

pppstats:  47,389,911 bytes in? 46,540 packets; 2,925,937 bytes out?

45,831 rxdma interrupts
658,825 SCC interrupts

2.6.27 pmac_zilog @ 4800:

pppstats:  10,653 bytes in? 145 packets; 13,089 bytes out? 195 packets
/proc/tty/driver/ttyPZ:  tx: 6,227; rx: 5,986 (!= pppstats???)

12,206 PowerMac Zilog interrupts

Interrupt load is higher without the DMA support.

Is it possible that this hardware was not meant to be used without the
DMA (i.e. it does not work quite right?)?

kevin

Re: pmac_zilog debugging ...

[Benjamin Herrenschmidt]

On Thu, 2008-11-13 at 03:38 -0800, Kevin Diggs wrote:

> 12,206 PowerMac Zilog interrupts
> 
> Interrupt load is higher without the DMA support.
> 
> Is it possible that this hardware was not meant to be used without the
> DMA (i.e. it does not work quite right?)?

Well, the HW Rx buffer is only 3 bytes so if you have high interrupt
latencies you are more likely to loose data...

Now, as I said, have you looked at flow control ? It's a likely cause of
problems and it's possible that pmac_zilog doesn't do it the way
macserial did...

Regarding DMA, it's possible to implement, though there were interesting
issues with the way it was done in macserial, it should be done
differently in pmac_zilog.

I think the only approach that really works properly (though it's fugly)
is what Apple does in OSX I think, which is to have a DMA descriptor per
input byte (no need for a huge DMA buffer anyway).

Cheers,
Ben.

Re: pmac_zilog debugging ...

[Kevin Diggs]

Benjamin Herrenschmidt wrote:
> On Thu, 2008-11-13 at 03:38 -0800, Kevin Diggs wrote:
> 
> 
>>12,206 PowerMac Zilog interrupts
>>
>>Interrupt load is higher without the DMA support.
>>
>>Is it possible that this hardware was not meant to be used without the
>>DMA (i.e. it does not work quite right?)?
> 
> 
> Well, the HW Rx buffer is only 3 bytes so if you have high interrupt
> latencies you are more likely to loose data...
> 
These are not real 8530s any more, right? How certain are we of this?
Is it possible that there is a larger buffer when used with the DMA
capability ... somehow?

> Now, as I said, have you looked at flow control ? It's a likely cause of
> problems and it's possible that pmac_zilog doesn't do it the way
> macserial did...
> 
I tried to put some debug statements where the flow lines are managed. I
could have goofed it up. They never produce any output. The latest
attempt used nortscts which should have disabled flow control. That
coupled with the fact that a 250 MHz 750GX is talking to a 486dx4 at
1200 - 9600 baud I would have thought would reduce the chance the PowerMac
would fall behind?

> Regarding DMA, it's possible to implement, though there were interesting
> issues with the way it was done in macserial, it should be done
> differently in pmac_zilog.
> 
> I think the only approach that really works properly (though it's fugly)
> is what Apple does in OSX I think, which is to have a DMA descriptor per
> input byte (no need for a huge DMA buffer anyway).
> 
You would need to explain to me the advantage of doing DMA in this case???

> Cheers,
> Ben.
> 
> 

Re: pmac_zilog debugging ...

[Benjamin Herrenschmidt]

On Thu, 2008-11-13 at 14:29 -0800, Kevin Diggs wrote:
> Benjamin Herrenschmidt wrote:
> > On Thu, 2008-11-13 at 03:38 -0800, Kevin Diggs wrote:
> > 
> > 
> >>12,206 PowerMac Zilog interrupts
> >>
> >>Interrupt load is higher without the DMA support.
> >>
> >>Is it possible that this hardware was not meant to be used without the
> >>DMA (i.e. it does not work quite right?)?
> > 
> > 
> > Well, the HW Rx buffer is only 3 bytes so if you have high interrupt
> > latencies you are more likely to loose data...
> > 
> These are not real 8530s any more, right? How certain are we of this?
> Is it possible that there is a larger buffer when used with the DMA
> capability ... somehow?

Well, the main thing is that when using DMA, it doesn't need to wait for
the kernel to come fetch the bytes, and thus the only latency that
matters if the DMA list is appropriately provisioned is the bus latency,
which is much less likely to be an issue even with a small buffer.

It's definitely not a "basic" 8530, it's an ESCC but I don't think the
base rx buffer in polled mode is any bigger (I may be wrong).

> I tried to put some debug statements where the flow lines are managed. I
> could have goofed it up. They never produce any output. The latest
> attempt used nortscts which should have disabled flow control. That
> coupled with the fact that a 250 MHz 750GX is talking to a 486dx4 at
> 1200 - 9600 baud I would have thought would reduce the chance the PowerMac
> would fall behind?

Have you disabled flow control both with the old macserial -and-
pmac_zilog and still experiencing the same problems ? (ie one works and
the other one doesn't ?)

> You would need to explain to me the advantage of doing DMA in this case???

Well, if I setup for example 128 DMA descriptors for 1 byte each, then
the chip will be able to DMA up to 128 bytes without CPU intervention,
thus is a -lot- less likely to overflow it's fifo. It's essentially a
way to have the DMA engine operate as an external FIFO. As the CPU
fetches the bytes, it can recycle the descriptors at the end of the list
effectively acting as some kind of ring buffer.

We can more easily do DMA for Tx but while this can improve performances
and lower interrupt usage, it is not a correctness issue in the sense
that Tx isn't -losing- data today, it's Rx that is a potential problem.

Cheers,
Ben.

Re: pmac_zilog debugging ...

[Kevin Diggs]

Benjamin Herrenschmidt wrote:
> On Thu, 2008-11-13 at 14:29 -0800, Kevin Diggs wrote:
> 
>>Benjamin Herrenschmidt wrote:
>>
>>>On Thu, 2008-11-13 at 03:38 -0800, Kevin Diggs wrote:
>>>
>>>
>>>
>>>>12,206 PowerMac Zilog interrupts
>>>>
>>>>Interrupt load is higher without the DMA support.
>>>>
>>>>Is it possible that this hardware was not meant to be used without the
>>>>DMA (i.e. it does not work quite right?)?
>>>
>>>
>>>Well, the HW Rx buffer is only 3 bytes so if you have high interrupt
>>>latencies you are more likely to loose data...
>>>
>>
>>These are not real 8530s any more, right? How certain are we of this?
>>Is it possible that there is a larger buffer when used with the DMA
>>capability ... somehow?
> 
> 
> Well, the main thing is that when using DMA, it doesn't need to wait for
> the kernel to come fetch the bytes, and thus the only latency that
> matters if the DMA list is appropriately provisioned is the bus latency,
> which is much less likely to be an issue even with a small buffer.
> 
"... if the DMA list is appropriately provisioned ..." ???

> It's definitely not a "basic" 8530, it's an ESCC but I don't think the
> base rx buffer in polled mode is any bigger (I may be wrong).
> 
Any idea how we might find out?
> 
>>I tried to put some debug statements where the flow lines are managed. I
>>could have goofed it up. They never produce any output. The latest
>>attempt used nortscts which should have disabled flow control. That
>>coupled with the fact that a 250 MHz 750GX is talking to a 486dx4 at
>>1200 - 9600 baud I would have thought would reduce the chance the PowerMac
>>would fall behind?
> 
> 
> Have you disabled flow control both with the old macserial -and-
> pmac_zilog and still experiencing the same problems ? (ie one works and
> the other one doesn't ?)
> 
I reinstalled the disk with 2.4.31 and reran the tests. With "nortscts" added to
the pppd options macserial still works fine. Even at 115,200 it seemed fine (I
expected to see some type of packet errors via pppstats or ifconfig). I did not,
however, verify that pppd actually does something with this option.

I can't get pmac_zilog to work even at 1200 baud. That is pretty slow.
> 
>>You would need to explain to me the advantage of doing DMA in this case???
> 
> 
> Well, if I setup for example 128 DMA descriptors for 1 byte each, then
> the chip will be able to DMA up to 128 bytes without CPU intervention,
> thus is a -lot- less likely to overflow it's fifo. It's essentially a
> way to have the DMA engine operate as an external FIFO. As the CPU
> fetches the bytes, it can recycle the descriptors at the end of the list
> effectively acting as some kind of ring buffer.
> 
> We can more easily do DMA for Tx but while this can improve performances
> and lower interrupt usage, it is not a correctness issue in the sense
> that Tx isn't -losing- data today, it's Rx that is a potential problem.
> 
Ah! I think I get it. I was thinking that cpu intervention would be required after
each byte. But the descriptors are more like linked commands for the DMA
hardware. So, I'm on board with this approach. Since I don't really know what I am
doing, how do you recommend I proceed?

Would it be correct to say that DMA would also free the cpu from doing io accesses
which are MUCH slower than normal memory acdcesses?

> Cheers,
> Ben.
> 
> 
> 

Re: pmac_zilog debugging ...

[Benjamin Herrenschmidt]

 
> > Well, the main thing is that when using DMA, it doesn't need to wait for
> > the kernel to come fetch the bytes, and thus the only latency that
> > matters if the DMA list is appropriately provisioned is the bus latency,
> > which is much less likely to be an issue even with a small buffer.
> > 
> "... if the DMA list is appropriately provisioned ..." ???

Well, if you have some descriptors to available buffers :-) If you let
that run out too, then the chip will miss

> > It's definitely not a "basic" 8530, it's an ESCC but I don't think the
> > base rx buffer in polled mode is any bigger (I may be wrong).
> > 
> Any idea how we might find out?

Nope. Trial and error ? :-)
 
> I reinstalled the disk with 2.4.31 and reran the tests. With "nortscts" added to
> the pppd options macserial still works fine. Even at 115,200 it seemed fine (I
> expected to see some type of packet errors via pppstats or ifconfig). I did not,
> however, verify that pppd actually does something with this option.
> 
> I can't get pmac_zilog to work even at 1200 baud. That is pretty slow.

That's definitely strange. I would expect the kernel to be able to get
interrupts fast enough to service a 1200 bauds serial port. Maybe
there's something else wrong, or an other driver causing undue interrupt
latencies.... 

Out of curiosity, check that IDE properly unmasks interrupts (hdparm
-u1 /dev/hda).

> Ah! I think I get it. I was thinking that cpu intervention would be required after
> each byte. But the descriptors are more like linked commands for the DMA
> hardware.

Yes.

>  So, I'm on board with this approach. Since I don't really know what I am
> doing, how do you recommend I proceed?

Google for a document called MacTech.pdf which contains various
documentations for bits of the ancestor of the IO chip in your machine,
along with a description of the DBDMA engine :-) Something else you can
do is to look at how it's properly used by other drivers such as bmac
and look at some of the darwin source code for reference on how the HW
works.

> Would it be correct to say that DMA would also free the cpu from doing io accesses
> which are MUCH slower than normal memory acdcesses?

To a certain extent yes.

Cheers,
Ben.

Re: pmac_zilog debugging ...

[Kevin Diggs]

Benjamin Herrenschmidt wrote:
>  
> 
> That's definitely strange. I would expect the kernel to be able to get
> interrupts fast enough to service a 1200 bauds serial port. Maybe
> there's something else wrong, or an other driver causing undue interrupt
> latencies.... 
> 
As far as I can see the system is NOT busy. I see no evidence of excessive
interrupt loading. It does have an Adaptec 2940 u2w SCSI card, an ATI video card,
and a USB/firewire card. The SCSI card has some disks on it. The other two cards
are unused. I guess, in theory, something in my 2.6.27 kernel could be causing one
of the two unused cards to throw spurious interrupts?

I still think the hardware is mis-behaving.

> Out of curiosity, check that IDE properly unmasks interrupts (hdparm
> -u1 /dev/hda).
> 
This is an 8600. It is SCSI only (the onboard controller is the MESH).
> 
>> So, I'm on board with this approach. Since I don't really know what I am
>>doing, how do you recommend I proceed?
> 
> 
> Google for a document called MacTech.pdf which contains various
> documentations for bits of the ancestor of the IO chip in your machine,
> along with a description of the DBDMA engine :-) Something else you can
> do is to look at how it's properly used by other drivers such as bmac
> and look at some of the darwin source code for reference on how the HW
> works.
> 
where might one find older Darwin source?

> Cheers,
> Ben.
> 
> _______________________________________________
> 

Re: pmac_zilog debugging ...

[Benjamin Herrenschmidt]

On Mon, 2008-11-17 at 02:21 -0800, Kevin Diggs wrote:
> Benjamin Herrenschmidt wrote:
> >  
> > 
> > That's definitely strange. I would expect the kernel to be able to get
> > interrupts fast enough to service a 1200 bauds serial port. Maybe
> > there's something else wrong, or an other driver causing undue interrupt
> > latencies.... 
> > 
> As far as I can see the system is NOT busy. I see no evidence of excessive
> interrupt loading. It does have an Adaptec 2940 u2w SCSI card, an ATI video card,
> and a USB/firewire card. The SCSI card has some disks on it. The other two cards
> are unused. I guess, in theory, something in my 2.6.27 kernel could be causing one
> of the two unused cards to throw spurious interrupts?
> 
> I still think the hardware is mis-behaving.

That's strange. Maybe one of the drivers is occasionally hogging
interrutps. Well, there may also be a bug in the code :-)

One thing you can try is to disable DMA in macserial (shouldn't be hard
to hack) and see if it degrades the same way.

> > Out of curiosity, check that IDE properly unmasks interrupts (hdparm
> > -u1 /dev/hda).
> > 
> This is an 8600. It is SCSI only (the onboard controller is the MESH).

Ah yes.
 
> >> So, I'm on board with this approach. Since I don't really know what I am
> >>doing, how do you recommend I proceed?
> > 
> > 
> > Google for a document called MacTech.pdf which contains various
> > documentations for bits of the ancestor of the IO chip in your machine,
> > along with a description of the DBDMA engine :-) Something else you can
> > do is to look at how it's properly used by other drivers such as bmac
> > and look at some of the darwin source code for reference on how the HW
> > works.
> > 
> where might one find older Darwin source?

Apple still has most of them back to 10.0 and even the recent ones still
have an SCC serial driver afaik.

Ben.

> > Cheers,
> > Ben.
> > 
> > _______________________________________________
> > 
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev

Re: pmac_zilog debugging ...

[Paul Mackerras]

Kevin Diggs writes:

> pppd ttyS0 1200 satellites: netmask 255.255.255.0 lock crtscts mru 1064 noauth debug kdebug 7 
> logfile /tmp/pppd.log local
> 
> to connect an 8600 to a laptop via ppp the link will lock up in short
> order from "payloaded" pings. Any advice on how to figure out where it
> is locking up? This command works fine to connect two x86 laptops. At
> 1200 it does take a while for an xterm to show up, though.

Try it without the crtscts (actually, use nocrtscts instead of
crtscts).  That will tell us whether it is hardware flow control
causing problems.  IIRC, those mac serial ports didn't have all of
RTS, CTS, CD and DTR, but I don't recall which one(s) were missing.

Also make sure you don't have the xonxoff option.

Paul.

Re: pmac_zilog debugging ...

[Benjamin Herrenschmidt]

On Sat, 2008-11-08 at 16:52 +1100, Paul Mackerras wrote:
> Kevin Diggs writes:
> 
> > pppd ttyS0 1200 satellites: netmask 255.255.255.0 lock crtscts mru 1064 noauth debug kdebug 7 
> > logfile /tmp/pppd.log local
> > 
> > to connect an 8600 to a laptop via ppp the link will lock up in short
> > order from "payloaded" pings. Any advice on how to figure out where it
> > is locking up? This command works fine to connect two x86 laptops. At
> > 1200 it does take a while for an xterm to show up, though.
> 
> Try it without the crtscts (actually, use nocrtscts instead of
> crtscts).  That will tell us whether it is hardware flow control
> causing problems.  IIRC, those mac serial ports didn't have all of
> RTS, CTS, CD and DTR, but I don't recall which one(s) were missing.
> 
> Also make sure you don't have the xonxoff option.

iirc, RTS is missing and on MacOS we typically have an option to use DTR
instead. There's also some issues with CTS polarity which is different
between the actual serial ports, vs. some internal modems or something
around those lines.

Ben