qmgr for ixp4xx

qmgr for ixp4xx

[Miguel Ángel Álvarez]

Hi

As I am trying to use my HSSs as 4E1, I find that the FIFOs for each
HDLC should be 1 word wide and not 4 words wide. I think that this
finally means that when accessing the queues, I have to do the
following modification.

diff -urN linux-2.6.26.7_hssorig/include/asm-arm/arch-ixp4xx/qmgr.h
linux-2.6.26.7/include/asm-arm/arch-ixp4xx/qmgr.h
--- linux-2.6.26.7_hssorig/include/asm-arm/arch-ixp4xx/qmgr.h	2008-10-22
23:46:18.000000000 +0200
+++ linux-2.6.26.7/include/asm-arm/arch-ixp4xx/qmgr.h	2008-12-02
10:47:23.000000000 +0100
@@ -67,16 +67,17 @@
 void qmgr_release_queue(unsigned int queue);


-static inline void qmgr_put_entry(unsigned int queue, u32 val)
+static inline void qmgr_put_entry(unsigned int queue, unsigned int pipe,
+                   u32 val)
 {
 	extern struct qmgr_regs __iomem *qmgr_regs;
-	__raw_writel(val, &qmgr_regs->acc[queue][0]);
+	__raw_writel(val, &qmgr_regs->acc[queue][pipe]);
 }

-static inline u32 qmgr_get_entry(unsigned int queue)
+static inline u32 qmgr_get_entry(unsigned int queue, unsigned int pipe)
 {
 	extern struct qmgr_regs __iomem *qmgr_regs;
-	return __raw_readl(&qmgr_regs->acc[queue][0]);
+	return __raw_readl(&qmgr_regs->acc[queue][pipe]);
 }

 static inline int qmgr_get_stat1(unsigned int queue)

Is it correct?

Thanks

Miguel Ángel Álvarez

Re: qmgr for ixp4xx

[Krzysztof Halasa]

"Miguel Ángel Álvarez" <gotzoncabanes@gmail.com> writes:

> As I am trying to use my HSSs as 4E1, I find that the FIFOs for each
> HDLC should be 1 word wide and not 4 words wide. I think that this
> finally means that when accessing the queues, I have to do the
> following modification.
>
> +++ linux-2.6.26.7/include/asm-arm/arch-ixp4xx/qmgr.h	2008-12-02
> 10:47:23.000000000 +0100
> @@ -67,16 +67,17 @@
>  void qmgr_release_queue(unsigned int queue);
>
>
> -static inline void qmgr_put_entry(unsigned int queue, u32 val)
> +static inline void qmgr_put_entry(unsigned int queue, unsigned int pipe,
> +                   u32 val)
>  {
>  	extern struct qmgr_regs __iomem *qmgr_regs;
> -	__raw_writel(val, &qmgr_regs->acc[queue][0]);
> +	__raw_writel(val, &qmgr_regs->acc[queue][pipe]);
>  }
>
> -static inline u32 qmgr_get_entry(unsigned int queue)
> +static inline u32 qmgr_get_entry(unsigned int queue, unsigned int pipe)
>  {
>  	extern struct qmgr_regs __iomem *qmgr_regs;
> -	return __raw_readl(&qmgr_regs->acc[queue][0]);
> +	return __raw_readl(&qmgr_regs->acc[queue][pipe]);
>  }

The FIFOs are some internal property of HDLC controller (it isn't
documented but they probably connect the bus master DMA controller to
the bit-stuffer and transmitter (and bit-destuffer and receiver in the
RX path)). You just need to send a message to HSS to tell it the
correct value.

Queues, on the other hand, can be 1, 2 or 4-words wide (32, 64 or 128
bits). I think nothing uses/needs 2 or 4-word queues so they aren't
implemented.

4E1 mode uses the same 1-word queues. It just uses more queues :-)
-- 
Krzysztof Halasa

Re: qmgr for ixp4xx

[Miguel Ángel Álvarez]

Hi

2008/12/4 Krzysztof Halasa <khc@pm.waw.pl>:
> "Miguel Ángel Álvarez" <gotzoncabanes@gmail.com> writes:
>
>> As I am trying to use my HSSs as 4E1, I find that the FIFOs for each
>> HDLC should be 1 word wide and not 4 words wide. I think that this
>> finally means that when accessing the queues, I have to do the
>> following modification.
>>
>> +++ linux-2.6.26.7/include/asm-arm/arch-ixp4xx/qmgr.h 2008-12-02
>> 10:47:23.000000000 +0100
>> @@ -67,16 +67,17 @@
>>  void qmgr_release_queue(unsigned int queue);
>>
>>
>> -static inline void qmgr_put_entry(unsigned int queue, u32 val)
>> +static inline void qmgr_put_entry(unsigned int queue, unsigned int pipe,
>> +                   u32 val)
>>  {
>>       extern struct qmgr_regs __iomem *qmgr_regs;
>> -     __raw_writel(val, &qmgr_regs->acc[queue][0]);
>> +     __raw_writel(val, &qmgr_regs->acc[queue][pipe]);
>>  }
>>
>> -static inline u32 qmgr_get_entry(unsigned int queue)
>> +static inline u32 qmgr_get_entry(unsigned int queue, unsigned int pipe)
>>  {
>>       extern struct qmgr_regs __iomem *qmgr_regs;
>> -     return __raw_readl(&qmgr_regs->acc[queue][0]);
>> +     return __raw_readl(&qmgr_regs->acc[queue][pipe]);
>>  }
>
> The FIFOs are some internal property of HDLC controller (it isn't
> documented but they probably connect the bus master DMA controller to
> the bit-stuffer and transmitter (and bit-destuffer and receiver in the
> RX path)). You just need to send a message to HSS to tell it the
> correct value.
>
¿The message is the HSS_CONFIG_CORE_CR containing the
CCR_NPE_HFIFO_3_OR_4HDLC flag?

> Queues, on the other hand, can be 1, 2 or 4-words wide (32, 64 or 128
> bits). I think nothing uses/needs 2 or 4-word queues so they aren't
> implemented.
>
> 4E1 mode uses the same 1-word queues. It just uses more queues :-)

I understand this, but I do not know exactly how to use it. I mean...
- I have seen that more queues are added for tx and rxfree, but not rx
or txdone... Are they not require?
- Must we use different txreadyq for each hdlc?
- The values you have chosen for txreadyq are 2 and 19. Does it not
conflict with HSS0_PKT_RXFREE1_QUEUE and HSS1_PKT_RXFREE1_QUEUE.
Shouldn't we use 23 and 24 (for example)?
- I am not sure which documentation did you use for this (great)
implementation of eth and hss. The intel manuals lack all information
about this, so I am trying to check differences with Intel's software
library (a nightmare), and have found that in "ixHssAccPktTxInternal"
they add the hdlc port to the entry (which at last seemed to me that
they were accessing consecutive memory positions from the entry for
hdlc0). So that is why I thought acc represented the same thing. If
not... What you are saying is that if I send the data to the correct
tx queue, it will reach the correct FIFO... OK. And reception? If I
have only one reception queue how could I manage to know to which
channel have I received the data?
- Do you know where can it be found more information about this
relation between queues, dmas and FIFOs? You are helping me greatly,
but I do not want to make you loose so much time.

Thanks

Miguel Ángel Álvarez

Re: qmgr for ixp4xx

[Krzysztof Halasa]

"Miguel Ángel Álvarez" <gotzoncabanes@gmail.com> writes:

>> The FIFOs are some internal property of HDLC controller (it isn't
>> documented but they probably connect the bus master DMA controller to
>> the bit-stuffer and transmitter (and bit-destuffer and receiver in the
>> RX path)). You just need to send a message to HSS to tell it the
>> correct value.
>>
> ¿The message is the HSS_CONFIG_CORE_CR containing the
> CCR_NPE_HFIFO_3_OR_4HDLC flag?

Well right, this one, too. I missed this one.

So it seems the following are needed (and 128-bit LUT):

#define PKT_NUM_PIPES           1 /* 1, 2 or 4 */
#define PKT_NUM_PIPES_WRITE             0x52

#define PKT_PIPE_FIFO_SIZEW     4 /* total 4 dwords per HSS */
(= 1 word for 4E1, 2 words for 2E1, 4 for 1E1)
#define PKT_PIPE_FIFO_SIZEW_WRITE       0x53

#define CCR_NPE_HFIFO_2_HDLC            0x04000000
#define CCR_NPE_HFIFO_3_OR_4HDLC        0x08000000
(to be set in the CORE register).

> I understand this, but I do not know exactly how to use it. I mean...
> - I have seen that more queues are added for tx and rxfree, but not rx
> or txdone... Are they not require?

The HSS uses 4 TX queues (each for 1 HDLC/packetized stream), and it
sends all used descriptors back to TXDONE queue (all 4 streams).

The same with RX, you have 4 RXFREE queues (= "RX empty" descriptors,
waiting for RX data), but when the data is ready, they are going back
to the CPU in a single RX queue. There is a stream# in the descriptor.

> - Must we use different txreadyq for each hdlc?

Yes, otherwise one HSS could grab all descriptors thus making the
other HSS (temporarily) unusable. Actually we need a TXREADY queues
per HDLC stream (4 per 4E1 port).

> - The values you have chosen for txreadyq are 2 and 19. Does it not
> conflict with HSS0_PKT_RXFREE1_QUEUE and HSS1_PKT_RXFREE1_QUEUE.

It does certainly conflict. For now there are no problems because MVIP
isn't supported. I guess we need 64-queue support. Fortunately, Karl
Hiramoto already has a patch for 64 queues, almost ready for merge.

We also have to make sure the queues don't conflict with the Ethernet
driver and (if used) with the crypto code.

> - I am not sure which documentation did you use for this (great)
> implementation of eth and hss. The intel manuals lack all information
> about this, so I am trying to check differences with Intel's software
> library (a nightmare),

I was using the library, though I processed it first with some custom
scripts to make it easier to read. Christian Hohnstaedt's code was
also a great help to understand what's going on.

> and have found that in "ixHssAccPktTxInternal"
> they add the hdlc port to the entry (which at last seemed to me that
> they were accessing consecutive memory positions from the entry for
> hdlc0).

Do you mean this?
/* Set the least significant 2 bits of the queue entry to the HDLC port number. */
qEntry |= (hdlcPortId & IX_HSSACC_QM_Q_CHAN_NUM_MASK);

They want it when the packet is back in TXDONE queue.

> So that is why I thought acc represented the same thing.

No, acc is a private thing of queue manager. HSS code don't know about
it.

> What you are saying is that if I send the data to the correct
> tx queue, it will reach the correct FIFO... OK.

Yes.

> And reception? If I
> have only one reception queue how could I manage to know to which
> channel have I received the data?

The descriptor will have an ID in it.

> - Do you know where can it be found more information about this
> relation between queues, dmas and FIFOs? You are helping me greatly,
> but I do not want to make you loose so much time.

We only have ixp4xx library code. Few small minutes of my time are
not a problem when spent to help implement a useful feature.

I wonder what should I do now with all this. Perhaps integrating
Karl's 64-queue patch and making the HDLC part of HSS driver able to
be merged upstream... I will look if it does make sense.
-- 
Krzysztof Halasa

Re: qmgr for ixp4xx

[Miguel Ángel Álvarez]

Hi

On Fri, Dec 5, 2008 at 5:03 PM, Krzysztof Halasa <khc@pm.waw.pl> wrote:
> "Miguel Ángel Álvarez" <gotzoncabanes@gmail.com> writes:
>
>>> The FIFOs are some internal property of HDLC controller (it isn't
>>> documented but they probably connect the bus master DMA controller to
>>> the bit-stuffer and transmitter (and bit-destuffer and receiver in the
>>> RX path)). You just need to send a message to HSS to tell it the
>>> correct value.
>>>
>> ¿The message is the HSS_CONFIG_CORE_CR containing the
>> CCR_NPE_HFIFO_3_OR_4HDLC flag?
>
> Well right, this one, too. I missed this one.
>
> So it seems the following are needed (and 128-bit LUT):
>
> #define PKT_NUM_PIPES           1 /* 1, 2 or 4 */
> #define PKT_NUM_PIPES_WRITE             0x52
>
> #define PKT_PIPE_FIFO_SIZEW     4 /* total 4 dwords per HSS */
> (= 1 word for 4E1, 2 words for 2E1, 4 for 1E1)
> #define PKT_PIPE_FIFO_SIZEW_WRITE       0x53
>
> #define CCR_NPE_HFIFO_2_HDLC            0x04000000
> #define CCR_NPE_HFIFO_3_OR_4HDLC        0x08000000
> (to be set in the CORE register).
>

Perfect... The ones I were using.

>> I understand this, but I do not know exactly how to use it. I mean...
>> - I have seen that more queues are added for tx and rxfree, but not rx
>> or txdone... Are they not require?
>
> The HSS uses 4 TX queues (each for 1 HDLC/packetized stream), and it
> sends all used descriptors back to TXDONE queue (all 4 streams).
>
> The same with RX, you have 4 RXFREE queues (= "RX empty" descriptors,
> waiting for RX data), but when the data is ready, they are going back
> to the CPU in a single RX queue. There is a stream# in the descriptor.
>
OK.

>> - Must we use different txreadyq for each hdlc?
>
> Yes, otherwise one HSS could grab all descriptors thus making the
> other HSS (temporarily) unusable. Actually we need a TXREADY queues
> per HDLC stream (4 per 4E1 port).
>
>> - The values you have chosen for txreadyq are 2 and 19. Does it not
>> conflict with HSS0_PKT_RXFREE1_QUEUE and HSS1_PKT_RXFREE1_QUEUE.
>
> It does certainly conflict. For now there are no problems because MVIP
> isn't supported. I guess we need 64-queue support. Fortunately, Karl
> Hiramoto already has a patch for 64 queues, almost ready for merge.
>
> We also have to make sure the queues don't conflict with the Ethernet
> driver and (if used) with the crypto code.
>
The queues are different for each NPE, aren' t they?

I will check for the 64-queue support patch (thanks Karl). However, if
I am not sure they are required. I mean...
- HSS0 uses queues 12-22.
- HSS1 uses queues 0-10.
- That leaves us 10 queues free, doesn't it? Couldn't we use queue 11
for eth txreadyq, 23-26 for HSS0 txreadyq, 27-30 for HSS1 txreadyq and
31 for crypto code (which I do not know?)

>> - I am not sure which documentation did you use for this (great)
>> implementation of eth and hss. The intel manuals lack all information
>> about this, so I am trying to check differences with Intel's software
>> library (a nightmare),
>
> I was using the library, though I processed it first with some custom
> scripts to make it easier to read. Christian Hohnstaedt's code was
> also a great help to understand what's going on.
>
I am trying to follow Intel's code... and it is difficult...

>> and have found that in "ixHssAccPktTxInternal"
>> they add the hdlc port to the entry (which at last seemed to me that
>> they were accessing consecutive memory positions from the entry for
>> hdlc0).
>
> Do you mean this?
> /* Set the least significant 2 bits of the queue entry to the HDLC port number. */
> qEntry |= (hdlcPortId & IX_HSSACC_QM_Q_CHAN_NUM_MASK);
>
> They want it when the packet is back in TXDONE queue.
>
Ummm? To know which queue should be refeeded? How could we do that?

>> So that is why I thought acc represented the same thing.
>
> No, acc is a private thing of queue manager. HSS code don't know about
> it.
>
OK.

>> What you are saying is that if I send the data to the correct
>> tx queue, it will reach the correct FIFO... OK.
>
> Yes.
>
Perfect.

>> And reception? If I
>> have only one reception queue how could I manage to know to which
>> channel have I received the data?
>
> The descriptor will have an ID in it.
>
Where?

>> - Do you know where can it be found more information about this
>> relation between queues, dmas and FIFOs? You are helping me greatly,
>> but I do not want to make you loose so much time.
>
> We only have ixp4xx library code. Few small minutes of my time are
> not a problem when spent to help implement a useful feature.
>
> I wonder what should I do now with all this. Perhaps integrating
> Karl's 64-queue patch and making the HDLC part of HSS driver able to
> be merged upstream... I will look if it does make sense.
>

I think your code is quite useful and structured, so it will be very
interesting to have it mainstream.

I was trying to improve your code to make it work in a 4E1. Before
making it configurable, I have made a first implementation considering
only my problem (which is not completely fixed yet). I will send you a
patch with my development (too large to add it in the list), so that
you could give me your oppinion if you want to. I would like to merge
it with your code when it is working (and the clean-up is made).

Thanks

Miguel Ángel

Re: qmgr for ixp4xx

[Krzysztof Halasa]

"Miguel Ángel Álvarez" <gotzoncabanes@gmail.com> writes:

>> We also have to make sure the queues don't conflict with the Ethernet
>> driver and (if used) with the crypto code.
>>
> The queues are different for each NPE, aren' t they?

They have to (except one Ethernet queue (TXDONE) which is used by all
ports).

> I will check for the 64-queue support patch (thanks Karl). However, if
> I am not sure they are required. I mean...
> - HSS0 uses queues 12-22.
> - HSS1 uses queues 0-10.
> - That leaves us 10 queues free, doesn't it? Couldn't we use queue 11
> for eth txreadyq, 23-26 for HSS0 txreadyq, 27-30 for HSS1 txreadyq and
> 31 for crypto code (which I do not know?)

Ethernet needs 3 queues per port + 1 (465 CPUs can have 3 Ethernet
ports), the crypto code probably needs several ones.

>> /* Set the least significant 2 bits of the queue entry to the HDLC port number. */
>> qEntry |= (hdlcPortId & IX_HSSACC_QM_Q_CHAN_NUM_MASK);
>>
>> They want it when the packet is back in TXDONE queue.
>>
> Ummm? To know which queue should be refeeded? How could we do that?

I think so. You'd have to check with the real hardware, I remember
some NPEs set some fields in the descriptors.

>>> And reception? If I
>>> have only one reception queue how could I manage to know to which
>>> channel have I received the data?
>>
>> The descriptor will have an ID in it.
>>
> Where?

In the lower address bits (bits 0-4?), like in the line above.

> I was trying to improve your code to make it work in a 4E1. Before
> making it configurable, I have made a first implementation considering
> only my problem (which is not completely fixed yet). I will send you a
> patch with my development (too large to add it in the list), so that
> you could give me your oppinion if you want to. I would like to merge
> it with your code when it is working (and the clean-up is made).

Great.
-- 
Krzysztof Halasa

Re: qmgr for ixp4xx

[Miguel Ángel Álvarez]

Hi

2008/12/5 Krzysztof Halasa <khc@pm.waw.pl>:
> "Miguel Ángel Álvarez" <gotzoncabanes@gmail.com> writes:
>
>>> We also have to make sure the queues don't conflict with the Ethernet
>>> driver and (if used) with the crypto code.
>>>
>> The queues are different for each NPE, aren' t they?
>
> They have to (except one Ethernet queue (TXDONE) which is used by all
> ports).
>
OK.

>> I will check for the 64-queue support patch (thanks Karl). However, if
>> I am not sure they are required. I mean...
>> - HSS0 uses queues 12-22.
>> - HSS1 uses queues 0-10.
>> - That leaves us 10 queues free, doesn't it? Couldn't we use queue 11
>> for eth txreadyq, 23-26 for HSS0 txreadyq, 27-30 for HSS1 txreadyq and
>> 31 for crypto code (which I do not know?)
>
> Ethernet needs 3 queues per port + 1 (465 CPUs can have 3 Ethernet
> ports), the crypto code probably needs several ones.
>
Sure... But in case you are not using ethernet in the NPEA (most
likely if you are using HSS), ethernet only takes one common queue
(TXDONE = 31). The rest of the queues seem to be NPE dependent. At
least that was what I thought, but if I change rxq and txreadyq for
ethernet in ixdp425-setup.c and give them the values 0x103, 0x114 for
NPEB and 0x203, 0x214 for NPEC they also interfere with the queues for
HSS (for not 0 hdlcs...).

So... Do we have 32 queues for NPE, or some of them are common for all
NPEs? In the first case it could be possible to work with 32 queues in
most of the situations... if not, 64-queue patch seems to be required
(where can I find last version of it? I just found one which is
heavily critizised by you, Krzysztof).

>>> /* Set the least significant 2 bits of the queue entry to the HDLC port number. */
>>> qEntry |= (hdlcPortId & IX_HSSACC_QM_Q_CHAN_NUM_MASK);
>>>
>>> They want it when the packet is back in TXDONE queue.
>>>
>> Ummm? To know which queue should be refeeded? How could we do that?
>
> I think so. You'd have to check with the real hardware, I remember
> some NPEs set some fields in the descriptors.
>
How can I check it?

>>>> And reception? If I
>>>> have only one reception queue how could I manage to know to which
>>>> channel have I received the data?
>>>
>>> The descriptor will have an ID in it.
>>>
>> Where?
>
> In the lower address bits (bits 0-4?), like in the line above.
>
OK.

Thanks

Miguel Ángel

Re: qmgr for ixp4xx

[Christian Hohnstaedt]

On Fri, Dec 05, 2008 at 07:29:36PM +0100, Krzysztof Halasa wrote:
> "Miguel ?ngel ?lvarez" <gotzoncabanes@gmail.com> writes:
> 
> > I will check for the 64-queue support patch (thanks Karl). However, if
> > I am not sure they are required. I mean...
> > - HSS0 uses queues 12-22.
> > - HSS1 uses queues 0-10.
> > - That leaves us 10 queues free, doesn't it? Couldn't we use queue 11
> > for eth txreadyq, 23-26 for HSS0 txreadyq, 27-30 for HSS1 txreadyq and
> > 31 for crypto code (which I do not know?)
> 
> Ethernet needs 3 queues per port + 1 (465 CPUs can have 3 Ethernet
> ports), the crypto code probably needs several ones.

Crypto needs 2 queues: 29 and 30.
The numbers are fixed in the microcode, I think.

Christian

Re: qmgr for ixp4xx

[Krzysztof Halasa]

"Miguel Ángel Álvarez" <gotzoncabanes@gmail.com> writes:

> Sure... But in case you are not using ethernet in the NPEA (most
> likely if you are using HSS),

I'm not sure but it may be true.
The docs say things about "6 ports" but AFAIK there are 3 physical
ports. For now I don't have anything 465-bases so I care a bit less
than for 425.

> ethernet only takes one common queue
> (TXDONE = 31). The rest of the queues seem to be NPE dependent.

Some numbers are fixed in software (like most of the Ethernet's
excluding "txready" which is a driver-only thing, and as Christian
writes, a couple of queues for crypto). Some are configurable (like RX
Ethernet queues, up to 8 per port with (not unused currently) QoS
"acceleration").

> At
> least that was what I thought, but if I change rxq and txreadyq for
> ethernet in ixdp425-setup.c and give them the values 0x103, 0x114 for
> NPEB and 0x203, 0x214 for NPEC they also interfere with the queues for
> HSS (for not 0 hdlcs...).

The range is 0-31 (0-0x1F). There are (not yet supported) queues 32-64
(0x20-0x3F) with a bit more limited functionality. We certainly need
them, too.

And the on-chip SRAM region in which the queues exist is limited, too.

> So... Do we have 32 queues for NPE, or some of them are common for all
> NPEs? In the first case it could be possible to work with 32 queues in
> most of the situations... if not, 64-queue patch seems to be required
> (where can I find last version of it? I just found one which is
> heavily critizised by you, Krzysztof).

It's this one. It wasn't "heavily criticized", though. I didn't like
many tiny bits but generally it is a good work.

We have 64 queues for all ports, processes etc combined.
You may want to see Intel's docs, for instance their IXP42X
development manual (the queues are explained in section 21, "AHB Queue
Manager (AQM)).
-- 
Krzysztof Halasa

Re: qmgr for ixp4xx

[Miguel Ángel Álvarez]

Hi

On Tue, Dec 9, 2008 at 5:44 PM, Christian Hohnstaedt
<christian@hohnstaedt.de> wrote:
> On Fri, Dec 05, 2008 at 07:29:36PM +0100, Krzysztof Halasa wrote:
>> "Miguel ?ngel ?lvarez" <gotzoncabanes@gmail.com> writes:
>>
>> > I will check for the 64-queue support patch (thanks Karl). However, if
>> > I am not sure they are required. I mean...
>> > - HSS0 uses queues 12-22.
>> > - HSS1 uses queues 0-10.
>> > - That leaves us 10 queues free, doesn't it? Couldn't we use queue 11
>> > for eth txreadyq, 23-26 for HSS0 txreadyq, 27-30 for HSS1 txreadyq and
>> > 31 for crypto code (which I do not know?)
>>
>> Ethernet needs 3 queues per port + 1 (465 CPUs can have 3 Ethernet
>> ports), the crypto code probably needs several ones.
>
> Crypto needs 2 queues: 29 and 30.
> The numbers are fixed in the microcode, I think.
>
Thanks for the info, Christian.

Miguel Ángel Álvarez

Re: qmgr for ixp4xx

[Miguel Ángel Álvarez]

Hi

2008/12/10 Krzysztof Halasa <khc@pm.waw.pl>:
> "Miguel Ángel Álvarez" <gotzoncabanes@gmail.com> writes:
>
>> Sure... But in case you are not using ethernet in the NPEA (most
>> likely if you are using HSS),
>
> I'm not sure but it may be true.
> The docs say things about "6 ports" but AFAIK there are 3 physical
> ports. For now I don't have anything 465-bases so I care a bit less
> than for 425.
>
ixp465 has only 3 ethernet ports I think. NPEA, NPEB and NPEC.

>> ethernet only takes one common queue
>> (TXDONE = 31). The rest of the queues seem to be NPE dependent.
>
> Some numbers are fixed in software (like most of the Ethernet's
> excluding "txready" which is a driver-only thing, and as Christian
> writes, a couple of queues for crypto). Some are configurable (like RX
> Ethernet queues, up to 8 per port with (not unused currently) QoS
> "acceleration").
>
Wouldn't it be interesting to have a list of all the fixed queues?
Let me try:
0-10 HSS1 (fixed)
12-22 HSS0 (fixed)
23 TXqueue NPEA (fixed)
24 TXqueue NPEB (fixed)
25 TXqueue NPEC (fixed)
26 RXfreequeue NPEA (fixed)
27 RXfreequeue NPEB (fixed)
28 RXfreequeue NPEC (fixed)
29,30 Criptocode (fixed)
31 Ethernet TXDONE (fixed)

Now supposing that we want to use three ethernets, two HSS and the
Criptocode, we need two additional queues for each ethernet (rxq and
txreadyq), and 2*4 additional queues for txreadyq for the possible 8
HDLC (4 per HSS) channels.

Am I right?

If we add 64 queue patch, we could give a fixed queue for each of this needs.

If not, in my case that I do not use ethernet in NPEA, criptocode and
HSSX_CHL_RXTRIG_QUEUEs, I have the following spare queues: 10, 11, 12,
23, 26, 29 and 30 (true?). Ummm... seven queues for eight txreadyq...
bad!!!

>> At
>> least that was what I thought, but if I change rxq and txreadyq for
>> ethernet in ixdp425-setup.c and give them the values 0x103, 0x114 for
>> NPEB and 0x203, 0x214 for NPEC they also interfere with the queues for
>> HSS (for not 0 hdlcs...).
>
> The range is 0-31 (0-0x1F). There are (not yet supported) queues 32-64
> (0x20-0x3F) with a bit more limited functionality. We certainly need
> them, too.
>
> And the on-chip SRAM region in which the queues exist is limited, too.
>
You are right... I was thinking the queues were different for each NPE
but they are not.

>> So... Do we have 32 queues for NPE, or some of them are common for all
>> NPEs? In the first case it could be possible to work with 32 queues in
>> most of the situations... if not, 64-queue patch seems to be required
>> (where can I find last version of it? I just found one which is
>> heavily critizised by you, Krzysztof).
>
> It's this one. It wasn't "heavily criticized", though. I didn't like
> many tiny bits but generally it is a good work.
>
OK. Sorry for saying "heavily criticized". Just wanted to state that
it seems not good enough for you to be added in your tree.

> We have 64 queues for all ports, processes etc combined.
> You may want to see Intel's docs, for instance their IXP42X
> development manual (the queues are explained in section 21, "AHB Queue
> Manager (AQM)).
>
RTFM... I was so centered in the lack of information of NPEs that I
did not realized that the queue manager had a chapter in the
developer's manual. Sorry. I have read it and things seem a bit
clearer now.

Thanks

Miguel Ángel

Re: qmgr for ixp4xx

[Miguel Ángel Álvarez]

Hi.

Complementing the previous mail.

I have implemented 64 queues for ixp4xx (following Karl's patch).

My idea was to use queues 32-39 for my eight txreadyq.

As rxq and txreadyq from ethernet in NPEB and NPEC conflict with the
queues in the ixp4xx_hss driver, I have decided to use queues 40-43
for them. Solving a problem with qmgr_stat_overflow, I have been able
to configure the interface (ethernet in NPEB). However not even a ping
can be sent.

Thinking that the problem was using >= 32 queues, I decided to use
queues 23-26... With the same result. So queues 3, 4, 20 and 21 seem
to be fixed for ethernet. But they are also used for HSS!!!

Any ideas?

Miguel Ángel