* Re: [PATCHv3 00/14] drivers: mailbox: framework creation
@ 2013-04-27 4:51 ` Jassi Brar
0 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-04-27 4:51 UTC (permalink / raw)
To: Suman Anna
Cc: Loic PALLARDY, Jassi Brar, Ohad Ben-Cohen (ohad@wizery.com),
Stephen Rothwell, Andy Green (andy.green@linaro.org),
Russell King, Arnd Bergmann, Tony Lindgren, Greg Kroah-Hartman,
Linus Walleij, Rafael J. Wysocki, Linux Kernel Mailing List,
Omar Ramirez Luna (omar.ramirez@copitl.com),
linux-arm-kernel@lists.infradead.org
Hi Suman,
>> On 26 April 2013 03:59, Suman Anna <s-anna@ti.com> wrote:
>>> On 04/25/2013 12:20 AM, Jassi Brar wrote:
>> I never said no-buffering and I never said buffering should be in
>> controller drivers. In fact I don't remember ever objecting to how
>> buffering is done in TI's framework.
>> A controller could service only 1 request at a time so lets give it
>> just 1 at a time. Let the API handle the complexity of buffering.
>>
>
> Alright, guess this got lost in translation :). I interpreted based on
> the fact that you wanted to get rid of the size field from the
> mailbox_msg definition. Do you have a different mechanism in mind for
> the buffering compared to the present one?
>
Sure, a very simple but efficient one. I had started on pseudo code
implementation the day I first replied, but now I have real code with
the PL320 controller and the Highbank client converted to the API. All
that I say features in the new design. Polishing and documentation
will take just a few hours more. You could see end to end what I have
been talking about.
>
> OK, I didn't think of a no RTR interrupt-based controller. I would thing
> that such a controller is very rudimentary. I wonder if there are any
> controllers like this out there.
>
One of my controllers is like that :)
>>
>> BTW, TI's RX mechanism too seems broken for common API. Receiving
>> every few bytes via 'notify' mechanism is very inefficient. Imagine a
>> platform with no shared memory between co-processors and the local
>> wants to diagnose the remote by asking critical data at least KBs in
>> size.
>
> No shared memory between co-processors and a relatively slow wire
> transport is a bad architecture design to begin with.
>
IMHO it's only about private memory. Even if the controller transfers,
say, 10bytes/interrupt there could always be a requirement to read
some 1MB region of remote's private memory. And the same logic implies
that our TX too should be as fast as possible - the remote might need
its 1MB firmware over the link. So let us just try to serve all
designs rather than evaluate them :)
>> So when API has nothing to do with received packet and the controller
>> has to get rid of it asap so as to be able to receive the next, IMHO
>> there should be short-circuit from controller to client via the API.
>> No delay, no buffering of RX.
>
> The current TI design is based on the fact that we can get multiple
> messages on a single interrupt due to the h/w fifo and the driver takes
> care of the bottom-half. Leaving it to the client is putting a lot of
> faith in the client and doesn't scale to multiple clients. The client
> would have to perform mostly the same as the driver is doing - so this
> goes back to the base discussion point that we have - which is the lack
> of support for atomic_context receivers in the current code. I perceive
> this as an attribute of the controller/mailbox device itself rather than
> the client.
>
Sorry, I don't understand the concern about faith.
If the controller h/w absolutely can not tell the remote(sender) of a
received packet (as seems to be your case), its driver shouldn't even
try to demux the received messages. The client driver must know which
remotes could send it a message and how to discern them on the
platform. Some 'server' RX client is needed here.
If the controller could indeed map received packet onto remotes, then
ideally the controller driver should declare one (RX only) channel for
each such remote and demux packets onto them.
In either case, 'notify' mechanism is not necessary.
> I agree that all remote-ends will not
> be able to cope up intermixed requests, but isn't this again a
> controller architecture dependent?
>
I think it's more about remote's protocol implementation than
controller's architecture.
If tomorrow TI's remote firmware introduces a new set of critical
commands that may arrive only in a particular sequence, you'll find
yourself sharing a ride on our dinghy :)
And Andy already explained where we come from.
Regards,
-Jassi
^ permalink raw reply [flat|nested] 90+ messages in thread
* [PATCH 1/3] mailbox: rename pl320-ipc specific mailbox.h
2013-04-27 4:51 ` Jassi Brar
@ 2013-04-27 18:05 ` jaswinder.singh
-1 siblings, 0 replies; 90+ messages in thread
From: jaswinder.singh at linaro.org @ 2013-04-27 18:05 UTC (permalink / raw)
To: linux-arm-kernel
From: Suman Anna <s-anna@ti.com>
The patch 30058677 "ARM / highbank: add support for pl320 IPC"
added a pl320 IPC specific header file as a generic mailbox.h.
This file has been renamed appropriately to allow the
introduction of the generic mailbox API framework.
Signed-off-by: Suman Anna <s-anna@ti.com>
Cc: Mark Langsdorf <mark.langsdorf@calxeda.com>
Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
drivers/cpufreq/highbank-cpufreq.c | 2 +-
drivers/mailbox/pl320-ipc.c | 2 +-
include/linux/{mailbox.h => pl320-ipc.h} | 0
3 files changed, 2 insertions(+), 2 deletions(-)
rename include/linux/{mailbox.h => pl320-ipc.h} (100%)
diff --git a/drivers/cpufreq/highbank-cpufreq.c b/drivers/cpufreq/highbank-cpufreq.c
index b61b5a3..3118b87 100644
--- a/drivers/cpufreq/highbank-cpufreq.c
+++ b/drivers/cpufreq/highbank-cpufreq.c
@@ -19,7 +19,7 @@
#include <linux/cpu.h>
#include <linux/err.h>
#include <linux/of.h>
-#include <linux/mailbox.h>
+#include <linux/pl320-ipc.h>
#include <linux/platform_device.h>
#define HB_CPUFREQ_CHANGE_NOTE 0x80000001
diff --git a/drivers/mailbox/pl320-ipc.c b/drivers/mailbox/pl320-ipc.c
index d873cba..f3755e0 100644
--- a/drivers/mailbox/pl320-ipc.c
+++ b/drivers/mailbox/pl320-ipc.c
@@ -26,7 +26,7 @@
#include <linux/device.h>
#include <linux/amba/bus.h>
-#include <linux/mailbox.h>
+#include <linux/pl320-ipc.h>
#define IPCMxSOURCE(m) ((m) * 0x40)
#define IPCMxDSET(m) (((m) * 0x40) + 0x004)
diff --git a/include/linux/mailbox.h b/include/linux/pl320-ipc.h
similarity index 100%
rename from include/linux/mailbox.h
rename to include/linux/pl320-ipc.h
--
1.7.10.4
^ permalink raw reply related [flat|nested] 90+ messages in thread* [PATCH 1/3] mailbox: rename pl320-ipc specific mailbox.h
@ 2013-04-27 18:05 ` jaswinder.singh
0 siblings, 0 replies; 90+ messages in thread
From: jaswinder.singh @ 2013-04-27 18:05 UTC (permalink / raw)
To: s-anna, loic.pallardy
Cc: arnd, linux-kernel, linux-arm-kernel, andy.green, Mark Langsdorf,
Rafael J. Wysocki
From: Suman Anna <s-anna@ti.com>
The patch 30058677 "ARM / highbank: add support for pl320 IPC"
added a pl320 IPC specific header file as a generic mailbox.h.
This file has been renamed appropriately to allow the
introduction of the generic mailbox API framework.
Signed-off-by: Suman Anna <s-anna@ti.com>
Cc: Mark Langsdorf <mark.langsdorf@calxeda.com>
Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
drivers/cpufreq/highbank-cpufreq.c | 2 +-
drivers/mailbox/pl320-ipc.c | 2 +-
include/linux/{mailbox.h => pl320-ipc.h} | 0
3 files changed, 2 insertions(+), 2 deletions(-)
rename include/linux/{mailbox.h => pl320-ipc.h} (100%)
diff --git a/drivers/cpufreq/highbank-cpufreq.c b/drivers/cpufreq/highbank-cpufreq.c
index b61b5a3..3118b87 100644
--- a/drivers/cpufreq/highbank-cpufreq.c
+++ b/drivers/cpufreq/highbank-cpufreq.c
@@ -19,7 +19,7 @@
#include <linux/cpu.h>
#include <linux/err.h>
#include <linux/of.h>
-#include <linux/mailbox.h>
+#include <linux/pl320-ipc.h>
#include <linux/platform_device.h>
#define HB_CPUFREQ_CHANGE_NOTE 0x80000001
diff --git a/drivers/mailbox/pl320-ipc.c b/drivers/mailbox/pl320-ipc.c
index d873cba..f3755e0 100644
--- a/drivers/mailbox/pl320-ipc.c
+++ b/drivers/mailbox/pl320-ipc.c
@@ -26,7 +26,7 @@
#include <linux/device.h>
#include <linux/amba/bus.h>
-#include <linux/mailbox.h>
+#include <linux/pl320-ipc.h>
#define IPCMxSOURCE(m) ((m) * 0x40)
#define IPCMxDSET(m) (((m) * 0x40) + 0x004)
diff --git a/include/linux/mailbox.h b/include/linux/pl320-ipc.h
similarity index 100%
rename from include/linux/mailbox.h
rename to include/linux/pl320-ipc.h
--
1.7.10.4
^ permalink raw reply related [flat|nested] 90+ messages in thread* [PATCH 1/3] mailbox: rename pl320-ipc specific mailbox.h
2013-04-27 18:05 ` jaswinder.singh
@ 2013-04-29 12:46 ` Mark Langsdorf
-1 siblings, 0 replies; 90+ messages in thread
From: Mark Langsdorf @ 2013-04-29 12:46 UTC (permalink / raw)
To: linux-arm-kernel
On 04/27/2013 01:05 PM, jaswinder.singh at linaro.org wrote:
> From: Suman Anna <s-anna@ti.com>
>
> The patch 30058677 "ARM / highbank: add support for pl320 IPC"
> added a pl320 IPC specific header file as a generic mailbox.h.
> This file has been renamed appropriately to allow the
> introduction of the generic mailbox API framework.
>
> Signed-off-by: Suman Anna <s-anna@ti.com>
> Cc: Mark Langsdorf <mark.langsdorf@calxeda.com>
> Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> ---
> drivers/cpufreq/highbank-cpufreq.c | 2 +-
> drivers/mailbox/pl320-ipc.c | 2 +-
> include/linux/{mailbox.h => pl320-ipc.h} | 0
> 3 files changed, 2 insertions(+), 2 deletions(-)
> rename include/linux/{mailbox.h => pl320-ipc.h} (100%)
>
> diff --git a/drivers/cpufreq/highbank-cpufreq.c b/drivers/cpufreq/highbank-cpufreq.c
> index b61b5a3..3118b87 100644
> --- a/drivers/cpufreq/highbank-cpufreq.c
> +++ b/drivers/cpufreq/highbank-cpufreq.c
> @@ -19,7 +19,7 @@
> #include <linux/cpu.h>
> #include <linux/err.h>
> #include <linux/of.h>
> -#include <linux/mailbox.h>
> +#include <linux/pl320-ipc.h>
> #include <linux/platform_device.h>
>
> #define HB_CPUFREQ_CHANGE_NOTE 0x80000001
> diff --git a/drivers/mailbox/pl320-ipc.c b/drivers/mailbox/pl320-ipc.c
> index d873cba..f3755e0 100644
> --- a/drivers/mailbox/pl320-ipc.c
> +++ b/drivers/mailbox/pl320-ipc.c
> @@ -26,7 +26,7 @@
> #include <linux/device.h>
> #include <linux/amba/bus.h>
>
> -#include <linux/mailbox.h>
> +#include <linux/pl320-ipc.h>
>
> #define IPCMxSOURCE(m) ((m) * 0x40)
> #define IPCMxDSET(m) (((m) * 0x40) + 0x004)
> diff --git a/include/linux/mailbox.h b/include/linux/pl320-ipc.h
> similarity index 100%
> rename from include/linux/mailbox.h
> rename to include/linux/pl320-ipc.h
>
Acked-by: Mark Langsdorf <mark.langsdorf@calxeda.com>
^ permalink raw reply [flat|nested] 90+ messages in thread* Re: [PATCH 1/3] mailbox: rename pl320-ipc specific mailbox.h
@ 2013-04-29 12:46 ` Mark Langsdorf
0 siblings, 0 replies; 90+ messages in thread
From: Mark Langsdorf @ 2013-04-29 12:46 UTC (permalink / raw)
To: jaswinder.singh@linaro.org
Cc: s-anna@ti.com, loic.pallardy@st.com, arnd@arndb.de,
linux-kernel@vger.kernel.org,
linux-arm-kernel@lists.infradead.org, andy.green@linaro.org,
Rafael J. Wysocki
On 04/27/2013 01:05 PM, jaswinder.singh@linaro.org wrote:
> From: Suman Anna <s-anna@ti.com>
>
> The patch 30058677 "ARM / highbank: add support for pl320 IPC"
> added a pl320 IPC specific header file as a generic mailbox.h.
> This file has been renamed appropriately to allow the
> introduction of the generic mailbox API framework.
>
> Signed-off-by: Suman Anna <s-anna@ti.com>
> Cc: Mark Langsdorf <mark.langsdorf@calxeda.com>
> Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> ---
> drivers/cpufreq/highbank-cpufreq.c | 2 +-
> drivers/mailbox/pl320-ipc.c | 2 +-
> include/linux/{mailbox.h => pl320-ipc.h} | 0
> 3 files changed, 2 insertions(+), 2 deletions(-)
> rename include/linux/{mailbox.h => pl320-ipc.h} (100%)
>
> diff --git a/drivers/cpufreq/highbank-cpufreq.c b/drivers/cpufreq/highbank-cpufreq.c
> index b61b5a3..3118b87 100644
> --- a/drivers/cpufreq/highbank-cpufreq.c
> +++ b/drivers/cpufreq/highbank-cpufreq.c
> @@ -19,7 +19,7 @@
> #include <linux/cpu.h>
> #include <linux/err.h>
> #include <linux/of.h>
> -#include <linux/mailbox.h>
> +#include <linux/pl320-ipc.h>
> #include <linux/platform_device.h>
>
> #define HB_CPUFREQ_CHANGE_NOTE 0x80000001
> diff --git a/drivers/mailbox/pl320-ipc.c b/drivers/mailbox/pl320-ipc.c
> index d873cba..f3755e0 100644
> --- a/drivers/mailbox/pl320-ipc.c
> +++ b/drivers/mailbox/pl320-ipc.c
> @@ -26,7 +26,7 @@
> #include <linux/device.h>
> #include <linux/amba/bus.h>
>
> -#include <linux/mailbox.h>
> +#include <linux/pl320-ipc.h>
>
> #define IPCMxSOURCE(m) ((m) * 0x40)
> #define IPCMxDSET(m) (((m) * 0x40) + 0x004)
> diff --git a/include/linux/mailbox.h b/include/linux/pl320-ipc.h
> similarity index 100%
> rename from include/linux/mailbox.h
> rename to include/linux/pl320-ipc.h
>
Acked-by: Mark Langsdorf <mark.langsdorf@calxeda.com>
^ permalink raw reply [flat|nested] 90+ messages in thread
* [RFC 2/3] mailbox: Introduce a new common API
2013-04-27 4:51 ` Jassi Brar
(?)
(?)
@ 2013-04-27 18:14 ` jassisinghbrar
2013-05-04 2:20 ` Suman Anna
-1 siblings, 1 reply; 90+ messages in thread
From: jassisinghbrar @ 2013-04-27 18:14 UTC (permalink / raw)
To: s-anna, loic.pallardy
Cc: arnd, linux-kernel, linux-arm-kernel, andy.green, Jassi Brar
From: Jassi Brar <jaswinder.singh@linaro.org>
Introduce common framework for client/protocol drivers and
controller drivers of Inter-Processor-Communication (IPC).
Client driver developers should have a look at
include/linux/mailbox_client.h to understand the part of
the API exposed to client drivers.
Similarly controller driver developers should have a look
at include/linux/mailbox_controller.h
Signed-off-by: Jassi Brar <jaswinder.singh@linaro.org>
---
drivers/mailbox/Makefile | 4 +
drivers/mailbox/mailbox.c | 470 ++++++++++++++++++++++++++++++++++++
include/linux/mailbox.h | 20 ++
include/linux/mailbox_client.h | 88 +++++++
include/linux/mailbox_controller.h | 105 ++++++++
5 files changed, 687 insertions(+)
create mode 100644 drivers/mailbox/mailbox.c
create mode 100644 include/linux/mailbox.h
create mode 100644 include/linux/mailbox_client.h
create mode 100644 include/linux/mailbox_controller.h
diff --git a/drivers/mailbox/Makefile b/drivers/mailbox/Makefile
index 543ad6a..fefef7e 100644
--- a/drivers/mailbox/Makefile
+++ b/drivers/mailbox/Makefile
@@ -1 +1,5 @@
+# Generic MAILBOX API
+
+obj-$(CONFIG_MAILBOX) += mailbox.o
+
obj-$(CONFIG_PL320_MBOX) += pl320-ipc.o
diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
new file mode 100644
index 0000000..c5ec93e
--- /dev/null
+++ b/drivers/mailbox/mailbox.c
@@ -0,0 +1,470 @@
+/*
+ * Copyright (C) 2013 Linaro Ltd
+ * Author: Jaswinder Singh <jassisinghbrar@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/interrupt.h>
+#include <linux/spinlock.h>
+#include <linux/mutex.h>
+#include <linux/delay.h>
+#include <linux/slab.h>
+#include <linux/err.h>
+#include <linux/module.h>
+#include <linux/mailbox_client.h>
+#include <linux/mailbox_controller.h>
+
+/*
+ * The length of circular buffer for queuing messages from a client.
+ * 'msg_count' tracks the number of buffered messages while 'msg_free'
+ * is the index where the next message would be buffered.
+ * We shouldn't need it too big because every transferr is interrupt
+ * triggered and if we have lots of data to transfer, the interrupt
+ * latencies are going to be the bottleneck, not the buffer length.
+ * Besides, ipc_send_message could be called from atomic context and
+ * the client could also queue another message from the notifier 'txcb'
+ * of the last transfer done.
+ */
+#define MBOX_TX_QUEUE_LEN 10
+
+#define TXDONE_BY_IRQ (1 << 0) /* controller has remote RTR irq */
+#define TXDONE_BY_POLL (1 << 1) /* controller can read status of last TX */
+#define TXDONE_BY_ACK (1 << 2) /* S/W ACK recevied by Client ticks the TX */
+
+struct ipc_chan {
+ char chan_name[32]; /* controller_name:link_name */
+ unsigned txdone_method;
+
+ /* Cached values from controller */
+ struct ipc_link *link;
+ struct ipc_link_ops *link_ops;
+
+ /* Cached values from client */
+ void (*rxcb)(void *data);
+ void (*txcb)(request_token_t t, enum xfer_result r);
+ bool tx_block;
+ unsigned long tx_tout;
+ struct completion tx_complete;
+
+ request_token_t active_token;
+ unsigned msg_count, msg_free;
+ void *msg_data[MBOX_TX_QUEUE_LEN];
+ /* Timer shared by all links of a controller */
+ struct tx_poll_timer *timer;
+ bool assigned;
+ /* Serialize access to the channel */
+ spinlock_t lock;
+ /* Hook to add to the global list of channels */
+ struct list_head node;
+ /* Notifier to all clients waiting on aquiring this channel */
+ struct blocking_notifier_head avail;
+};
+
+/*
+ * If the controller supports only TXDONE_BY_POLL, this
+ * timer polls all the links for txdone.
+ */
+struct tx_poll_timer {
+ struct timer_list poll;
+ unsigned period;
+};
+
+static struct list_head ipc_channels;
+static DEFINE_MUTEX(chpool_mutex);
+
+static request_token_t _add_to_rbuf(struct ipc_chan *chan, void *data)
+{
+ request_token_t idx;
+ unsigned long flags;
+
+ spin_lock_irqsave(&chan->lock, flags);
+
+ /* See if there is any space left */
+ if (chan->msg_count == MBOX_TX_QUEUE_LEN) {
+ spin_unlock_irqrestore(&chan->lock, flags);
+ return 0;
+ }
+
+ idx = chan->msg_free;
+ chan->msg_data[idx] = data;
+ chan->msg_count++;
+
+ if (idx == MBOX_TX_QUEUE_LEN - 1)
+ chan->msg_free = 0;
+ else
+ chan->msg_free++;
+
+ spin_unlock_irqrestore(&chan->lock, flags);
+
+ return idx + 1;
+}
+
+static void _msg_submit(struct ipc_chan *chan)
+{
+ struct ipc_link *link = chan->link;
+ unsigned count, idx;
+ unsigned long flags;
+ void *data;
+ int err;
+
+ spin_lock_irqsave(&chan->lock, flags);
+
+ if (!chan->msg_count || chan->active_token) {
+ spin_unlock_irqrestore(&chan->lock, flags);
+ return;
+ }
+
+ count = chan->msg_count;
+ idx = chan->msg_free;
+ if (idx >= count)
+ idx -= count;
+ else
+ idx += MBOX_TX_QUEUE_LEN - count;
+
+ data = chan->msg_data[idx];
+
+ /* Try to submit a message to the IPC controller */
+ err = chan->link_ops->send_data(link, data);
+ if (!err) {
+ chan->active_token = idx + 1;
+ chan->msg_count--;
+ }
+
+ spin_unlock_irqrestore(&chan->lock, flags);
+}
+
+static void tx_tick(struct ipc_chan *chan, enum xfer_result r)
+{
+ unsigned long flags;
+ request_token_t t;
+
+ spin_lock_irqsave(&chan->lock, flags);
+ t = chan->active_token;
+ chan->active_token = 0;
+ spin_unlock_irqrestore(&chan->lock, flags);
+
+ /* Submit next message */
+ _msg_submit(chan);
+
+ /* Notify the client */
+ if (chan->tx_block)
+ complete(&chan->tx_complete);
+ else if (t && chan->txcb)
+ chan->txcb(t, r);
+}
+
+static void poll_txdone(unsigned long data)
+{
+ struct tx_poll_timer *timer = (struct tx_poll_timer *)data;
+ bool txdone, resched = false;
+ struct ipc_chan *chan;
+
+ list_for_each_entry(chan, &ipc_channels, node) {
+ if (chan->timer == timer && chan->active_token) {
+ resched = true;
+ txdone = chan->link_ops->last_tx_done(chan->link);
+ if (txdone)
+ tx_tick(chan, XFER_OK);
+ }
+ }
+
+ if (resched)
+ mod_timer(&timer->poll,
+ jiffies + msecs_to_jiffies(timer->period));
+}
+
+/*
+ * After 'startup' and before 'shutdown', the IPC controller driver
+ * notifies the API of data received over the link.
+ * The controller driver should make sure the 'RTR' is de-asserted since
+ * reception of the packet and until after this call returns.
+ * This call could be made from atomic context.
+ */
+void ipc_link_received_data(struct ipc_link *link, void *data)
+{
+ struct ipc_chan *chan = (struct ipc_chan *)link->api_priv;
+
+ /* No buffering the received data */
+ if (chan->rxcb)
+ chan->rxcb(data);
+}
+EXPORT_SYMBOL(ipc_link_received_data);
+
+/*
+ * The IPC controller driver notifies the API that the remote has
+ * asserted RTR and it could now send another message on the link.
+ */
+void ipc_link_txdone(struct ipc_link *link, enum xfer_result r)
+{
+ struct ipc_chan *chan = (struct ipc_chan *)link->api_priv;
+
+ if (unlikely(!(chan->txdone_method & TXDONE_BY_IRQ))) {
+ printk("Controller can't run the TX ticker\n");
+ return;
+ }
+
+ tx_tick(chan, r);
+}
+EXPORT_SYMBOL(ipc_link_txdone);
+
+/*
+ * The client/protocol had received some 'ACK' packet and it notifies
+ * the API that the last packet was sent successfully. This only works
+ * if the controller doesn't get IRQ for TX done.
+ */
+void ipc_client_txdone(void *channel, enum xfer_result r)
+{
+ struct ipc_chan *chan = (struct ipc_chan *)channel;
+ bool txdone = true;
+
+ if (unlikely(!(chan->txdone_method & TXDONE_BY_ACK))) {
+ printk("Client can't run the TX ticker\n");
+ return;
+ }
+
+ if (chan->txdone_method & TXDONE_BY_POLL)
+ txdone = chan->link_ops->last_tx_done(chan->link);
+
+ if (txdone)
+ tx_tick(chan, r);
+}
+EXPORT_SYMBOL(ipc_client_txdone);
+
+/*
+ * Called by a client to "put data on the h/w channel" so that if
+ * everything else is fine we don't need to do anything more locally
+ * for the remote to receive the data intact.
+ * In reality, the remote may receive it intact, corrupted or not at all.
+ * This could be called from atomic context as it simply
+ * queues the data and returns a token (request_token_t)
+ * against the request.
+ * The client is later notified of successful transmission of
+ * data over the channel via the 'txcb'. The client could in
+ * turn queue more messages from txcb.
+ */
+request_token_t ipc_send_message(void *channel, void *data)
+{
+ struct ipc_chan *chan = (struct ipc_chan *)channel;
+ request_token_t t;
+
+ if (!chan)
+ return 0;
+
+ t = _add_to_rbuf(chan, data);
+ if (!t)
+ printk("Try increasing MBOX_TX_QUEUE_LEN\n");
+
+ _msg_submit(chan);
+
+ if (chan->txdone_method == TXDONE_BY_POLL)
+ poll_txdone((unsigned long)chan->timer);
+
+ if (chan->tx_block && chan->active_token) {
+ int ret;
+ init_completion(&chan->tx_complete);
+ ret = wait_for_completion_timeout(&chan->tx_complete,
+ chan->tx_tout);
+ if (ret == 0) {
+ t = 0;
+ tx_tick(chan, XFER_ERR);
+ }
+ }
+
+ return t;
+}
+EXPORT_SYMBOL(ipc_send_message);
+
+/*
+ * A client driver asks for exclusive use of a channel/mailbox.
+ * If assigned, the channel has to be 'freed' before it could
+ * be assigned to some other client.
+ * After assignment, any packet received on this channel will be
+ * handed over to the client via the 'rxcb' callback.
+ * The 'txcb' callback is used to notify client upon sending the
+ * packet over the channel, which may or may not have been yet
+ * read by the remote processor.
+ */
+void *ipc_request_channel(struct ipc_client *cl)
+{
+ struct ipc_chan *chan;
+ unsigned long flags;
+ int ret = 0;
+
+ mutex_lock(&chpool_mutex);
+
+ list_for_each_entry(chan, &ipc_channels, node)
+ if(!chan->assigned
+ && !strcmp(cl->chan_name, chan->chan_name)) {
+ spin_lock_irqsave(&chan->lock, flags);
+ chan->msg_free = 0;
+ chan->msg_count = 0;
+ chan->active_token = 0;
+ chan->rxcb = cl->rxcb;
+ chan->txcb = cl->txcb;
+ chan->assigned = true;
+ chan->tx_block = cl->tx_block;
+ if (!cl->tx_tout)
+ chan->tx_tout = ~0;
+ else
+ chan->tx_tout = msecs_to_jiffies(cl->tx_tout);
+ if (chan->txdone_method == TXDONE_BY_POLL
+ && cl->knows_txdone)
+ chan->txdone_method |= TXDONE_BY_ACK;
+ spin_unlock_irqrestore(&chan->lock, flags);
+ ret = 1;
+ break;
+ }
+
+ mutex_unlock(&chpool_mutex);
+
+ if (!ret) {
+ printk("Unable to assign mailbox(%s)\n", cl->chan_name);
+ return NULL;
+ }
+
+ ret = chan->link_ops->startup(chan->link, cl->cntlr_data);
+ if (ret) {
+ printk("Unable to startup the link\n");
+ ipc_free_channel((void *)chan);
+ return NULL;
+ }
+
+ return (void *)chan;
+}
+EXPORT_SYMBOL(ipc_request_channel);
+
+/* Drop any messages queued and release the channel */
+void ipc_free_channel(void *ch)
+{
+ struct ipc_chan *chan = (struct ipc_chan *)ch;
+ unsigned long flags;
+
+ if (!chan || !chan->assigned)
+ return;
+
+ chan->link_ops->shutdown(chan->link);
+
+ /* The queued TX requests are simply aborted, no callbacks are made */
+ spin_lock_irqsave(&chan->lock, flags);
+ chan->assigned = false;
+ if (chan->txdone_method == (TXDONE_BY_POLL | TXDONE_BY_ACK))
+ chan->txdone_method = TXDONE_BY_POLL;
+ spin_unlock_irqrestore(&chan->lock, flags);
+
+ blocking_notifier_call_chain(&chan->avail, 0, NULL);
+}
+EXPORT_SYMBOL(ipc_free_channel);
+
+static struct ipc_chan *name_to_chan(const char *name)
+{
+ struct ipc_chan *chan;
+ int ret = 0;
+
+ mutex_lock(&chpool_mutex);
+ list_for_each_entry(chan, &ipc_channels, node)
+ if (!strcmp(name, chan->chan_name)) {
+ ret = 1;
+ break;
+ }
+ mutex_unlock(&chpool_mutex);
+
+ if (!ret)
+ return NULL;
+
+ return chan;
+}
+
+int ipc_notify_chan_register(const char *name, struct notifier_block *nb)
+{
+ struct ipc_chan *chan = name_to_chan(name);
+
+ if (chan && nb)
+ return blocking_notifier_chain_register(&chan->avail, nb);
+
+ return -EINVAL;
+}
+EXPORT_SYMBOL(ipc_notify_chan_register);
+
+void ipc_notify_chan_unregister(const char *name, struct notifier_block *nb)
+{
+ struct ipc_chan *chan = name_to_chan(name);
+
+ if (chan && nb)
+ blocking_notifier_chain_unregister(&chan->avail, nb);
+}
+EXPORT_SYMBOL(ipc_notify_chan_unregister);
+
+/*
+ * Call for IPC controller drivers to register a controller, adding
+ * its channels/mailboxes to the global pool.
+ */
+int ipc_links_register(struct ipc_controller *ipc_con)
+{
+ struct tx_poll_timer *timer = NULL;
+ struct ipc_chan *channel;
+ int i, num_links, txdone;
+
+ /* Are you f***ing with us, sir? */
+ if (!ipc_con || !ipc_con->ops)
+ return -EINVAL;
+
+ for (i = 0; ipc_con->links[i]; i++)
+ ;
+ if (!i)
+ return -EINVAL;
+ num_links = i;
+
+ if (ipc_con->txdone_irq)
+ txdone = TXDONE_BY_IRQ;
+ else if (ipc_con->txdone_poll)
+ txdone = TXDONE_BY_POLL;
+ else /* It has to be at least ACK */
+ txdone = TXDONE_BY_ACK;
+
+ if (txdone == TXDONE_BY_POLL) {
+ timer = kzalloc(sizeof(struct tx_poll_timer), GFP_KERNEL);
+ timer->period = ipc_con->txpoll_period;
+ timer->poll.function = &poll_txdone;
+ timer->poll.data = (unsigned long)timer;
+ init_timer(&timer->poll);
+ }
+
+ channel = kzalloc(sizeof(struct ipc_chan) * num_links, GFP_KERNEL);
+
+ for (i = 0; i < num_links; i++) {
+ channel[i].timer = timer;
+ channel[i].assigned = false;
+ channel[i].txdone_method = txdone;
+ channel[i].link_ops = ipc_con->ops;
+ channel[i].link = ipc_con->links[i];
+ channel[i].link->api_priv = &channel[i];
+ snprintf(channel[i].chan_name, 32, "%s:%s",
+ ipc_con->controller_name,
+ ipc_con->links[i]->link_name);
+ spin_lock_init(&channel[i].lock);
+ BLOCKING_INIT_NOTIFIER_HEAD(&channel[i].avail);
+ mutex_lock(&chpool_mutex);
+ list_add_tail(&channel[i].node, &ipc_channels);
+ mutex_unlock(&chpool_mutex);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(ipc_links_register);
+
+/* Free any occupied channels */
+void ipc_links_unregister(struct ipc_controller *ipc_con)
+{
+ /* TBD */
+}
+EXPORT_SYMBOL(ipc_links_unregister);
+
+static int __init ipc_mbox_init(void)
+{
+ INIT_LIST_HEAD(&ipc_channels);
+ return 0;
+}
+subsys_initcall(ipc_mbox_init);
diff --git a/include/linux/mailbox.h b/include/linux/mailbox.h
new file mode 100644
index 0000000..aa82be8
--- /dev/null
+++ b/include/linux/mailbox.h
@@ -0,0 +1,20 @@
+/*
+ * Copyright (C) 2013 Linaro Ltd
+ * Author: Jaswinder Singh <jassisinghbrar@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef __MAILBOX_H
+#define __MAILBOX_H
+
+enum xfer_result {
+ XFER_OK = 0,
+ XFER_ERR,
+};
+
+typedef unsigned request_token_t;
+
+#endif /* __MAILBOX_H */
diff --git a/include/linux/mailbox_client.h b/include/linux/mailbox_client.h
new file mode 100644
index 0000000..33545f6
--- /dev/null
+++ b/include/linux/mailbox_client.h
@@ -0,0 +1,88 @@
+/*
+ * Copyright (C) 2013 Linaro Ltd
+ * Author: Jaswinder Singh <jassisinghbrar@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef __MAILBOX_CLIENT_H
+#define __MAILBOX_CLIENT_H
+
+#include <linux/mailbox.h>
+
+/**
+ * struct ipc_client - User of a mailbox
+ * @chan_name: the "controller:channel" this client wants
+ * @rxcb: atomic callback to provide client the data received
+ * @txcb: atomic callback to tell client of data transmission
+ * @tx_block: if the ipc_send_message should block until data is transmitted
+ * @tx_tout: Max block period in ms before TX is assumed failure
+ * @knows_txdone: if the client could run the TX state machine. Usually if
+ * the client receives some ACK packet for transmission. Unused if the
+ * controller already has TX_Done/RTR IRQ.
+ * @cntlr_data: Optional controller specific parameters during channel request
+ */
+struct ipc_client {
+ char *chan_name;
+ void (*rxcb)(void *data);
+ void (*txcb)(request_token_t t, enum xfer_result r);
+ bool tx_block;
+ unsigned long tx_tout;
+ bool knows_txdone;
+ void *cntlr_data;
+};
+
+/**
+ * The Client specifies it requirements and capabilities while asking for
+ * a channel/mailbox by name. It can't be called from atomic context.
+ * The channel is exclusively allocated and can't be used by another
+ * client before the owner calls ipc_free_channel.
+ */
+void *ipc_request_channel(struct ipc_client *cl);
+
+/**
+ * For client to submit data to the controller destined for a remote
+ * processor. If the client had set 'tx_block', the call will return
+ * either when the remote receives the data or when 'tx_tout' millisecs
+ * run out.
+ * In non-blocking mode, the requests are buffered by the API and a
+ * non-zero token is returned for each queued request. If the queue
+ * was full the returned token will be 0. Upon failure or successful
+ * TX, the API calls 'txcb' from atomic context, from which the client
+ * could submit yet another request.
+ * In blocking mode, 'txcb' is not called, effectively making the
+ * queue length 1. The returned value is 0 if TX timed out, some
+ * non-zero value upon success.
+ */
+request_token_t ipc_send_message(void *channel, void *data);
+
+/**
+ * The way for a client to run the TX state machine. This works
+ * only if the client sets 'knows_txdone' and the IPC controller
+ * don't get an IRQ for TX_Done.
+ */
+void ipc_client_txdone(void *channel, enum xfer_result r);
+
+/**
+ * The client relinquishes control of a mailbox by this call,
+ * make it available to other clients.
+ * The ipc_request/free_channel are light weight calls, so the
+ * client should avoid holding it when it doesn't need to
+ * transfer data.
+ */
+void ipc_free_channel(void *ch);
+
+/**
+ * The client make ask the API to be notified when a particular channel
+ * becomes available to be acquired again.
+ */
+int ipc_notify_chan_register(const char *name, struct notifier_block *nb);
+
+/**
+ * The client is no more interested in acquiring the channel.
+ */
+void ipc_notify_chan_unregister(const char *name, struct notifier_block *nb);
+
+#endif /* __MAILBOX_CLIENT_H */
diff --git a/include/linux/mailbox_controller.h b/include/linux/mailbox_controller.h
new file mode 100644
index 0000000..defcfb3
--- /dev/null
+++ b/include/linux/mailbox_controller.h
@@ -0,0 +1,105 @@
+/*
+ * Copyright (C) 2013 Linaro Ltd
+ * Author: Jaswinder Singh <jassisinghbrar@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef __MAILBOX_CONTROLLER_H
+#define __MAILBOX_CONTROLLER_H
+
+#include <linux/mailbox.h>
+
+/**
+ * struct ipc_link - s/w representation of a communication link
+ * @link_name: Literal name assigned to the link. Physically
+ * identical channels may have the same name.
+ * @api_priv: hook for the API to map its private data on the link
+ * Controller driver must not touch it.
+ */
+struct ipc_link {
+ char link_name[16];
+ void *api_priv;
+};
+
+/**
+ * struct ipc_link - s/w representation of a communication link
+ * @send_data: The API asks the IPC controller driver, in atomic
+ * context try to transmit a message on the bus. Returns 0 if
+ * data is accepted for transmission, -EBUSY while rejecting
+ * if the remote hasn't yet read the last data sent. Actual
+ * transmission of data is reported by the controller via
+ * ipc_link_txdone (if it has some TX ACK irq). It must not
+ * block.
+ * @startup: Called when a client requests the link. The controller
+ * could ask clients for additional parameters of communication
+ * to be provided via client's cntlr_data. This call may block.
+ * After this call the Controller must forward any data received
+ * on the link by calling ipc_link_received_data (which won't block)
+ * @shutdown: Called when a client relinquishes control of a link.
+ * This call may block too. The controller must not forwared
+ * any received data anymore.
+ * @last_tx_done: If the controller sets 'txdone_poll', the API calls
+ * this to poll status of last TX. The controller must give priority
+ * to IRQ method over polling and never set both txdone_poll and
+ * txdone_irq. Only in polling mode 'send_data' is expected to
+ * return -EBUSY. Used only if txdone_poll:=true && txdone_irq:=false
+ */
+struct ipc_link_ops {
+ int (*send_data)(struct ipc_link *link, void *data);
+ int (*startup)(struct ipc_link *link, void *params);
+ void (*shutdown)(struct ipc_link *link);
+ bool (*last_tx_done)(struct ipc_link *link);
+};
+
+/**
+ * struct ipc_controller - Controller of a class of communication links
+ * @controller_name: Literal name of the controller.
+ * @ops: Operators that work on each communication link
+ * @links: Null terminated array of links.
+ * @txdone_irq: Indicates if the controller can report to API when the
+ * last transmitted data was read by the remote. Eg, if it has some
+ * TX ACK irq.
+ * @txdone_poll: If the controller can read but not report the TX done.
+ * Eg, is some register shows the TX status but no interrupt rises.
+ * Ignored if 'txdone_irq' is set.
+ * @txpoll_period: If 'txdone_poll' is in effect, the API polls for
+ * last TX's status after these many millisecs
+ */
+struct ipc_controller {
+ char controller_name[16];
+ struct ipc_link_ops *ops;
+ struct ipc_link **links;
+ bool txdone_irq;
+ bool txdone_poll;
+ unsigned txpoll_period;
+};
+
+/**
+ * The controller driver registers its communication links to the
+ * global pool managed by the API.
+ */
+int ipc_links_register(struct ipc_controller *ipc_con);
+
+/**
+ * After startup and before shutdown any data received on the link
+ * is pused to the API via atomic ipc_link_received_data() API.
+ * The controller should ACK the RX only after this call returns.
+ */
+void ipc_link_received_data(struct ipc_link *link, void *data);
+
+/**
+ * The controller the has IRQ for TX ACK calls this atomic API
+ * to tick the TX state machine. It works only if txdone_irq
+ * is set by the controller.
+ */
+void ipc_link_txdone(struct ipc_link *link, enum xfer_result r);
+
+/**
+ * Purge the links from the global pool maintained by the API.
+ */
+void ipc_links_unregister(struct ipc_controller *ipc_con);
+
+#endif /* __MAILBOX_CONTROLLER_H */
--
1.7.10.4
^ permalink raw reply related [flat|nested] 90+ messages in thread* [RFC 2/3] mailbox: Introduce a new common API
2013-04-27 18:14 ` [RFC 2/3] mailbox: Introduce a new common API jassisinghbrar
@ 2013-05-04 2:20 ` Suman Anna
0 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-05-04 2:20 UTC (permalink / raw)
To: linux-arm-kernel
Hi Jassi,
On 04/27/2013 01:14 PM, jassisinghbrar at gmail.com wrote:
> From: Jassi Brar <jaswinder.singh@linaro.org>
>
> Introduce common framework for client/protocol drivers and
> controller drivers of Inter-Processor-Communication (IPC).
>
> Client driver developers should have a look at
> include/linux/mailbox_client.h to understand the part of
> the API exposed to client drivers.
> Similarly controller driver developers should have a look
> at include/linux/mailbox_controller.h
This implementation looks decent based on your design points. These
are the open/contention points that needs to be sorted out.
I think understanding the OMAP mailbox architecture also helps you,
since this series currently addressed PL320, but we will run into
some issues when adopting for OMAP as is. OMAP has a single mailbox IP,
each with multiple h/w FIFOs (fifo of length 4, and message size of a
u32). Each IP can cause one or more irq (usually 1) into the linux host.
It has status registers for number of messages in a FIFO (Rx), FIFO full
(Tx non-readiness/busy state), and interrupts for both Rx and Tx. There
are registers indicating the source of the interrupt, and these are per
FIFO. The Tx interrupt source is really for indicating Tx readiness or
that a fifo has open room for sending messages. This will keep on firing
as long as the FIFO is not-full, so we usually configure this only when
the FIFO is full and disable it the moment it is fired. It is re-enabled
when the h/w FIFO is full again, and we use s/w buffering in between.
The Rx interrupt is fired as long as there is a message in any FIFO
which has been configured to interrupt the host, so we have to empty all
the messages for that interrupt source.
Anyway, here is a summary of the open points that we have:
1. Atomic Callbacks:
The current code provides some sort of buffering on Tx, but imposes the
restriction that the clients do the buffering on Rx. This is main
concern for OMAP. We have multiple links on a single interrupt and
multiple messages per link on a interrupt source. Pushing this logic
into a client (remoteproc for us) is not right, that code does not
belong to the remoteproc driver, and neither is adding another layer in
between. It also kills the shared clients idea. The current
ipc_link_received_data seems open enough for me to use by calling it
from a work-queue, and have the OMAP controller driver take care of Rx
buffering (will test this next week). Restricting ipc_link_received_data
to atomic callbacks might have been enough for single transport
payloads, but we do have a FIFO. I could possibly make it work using
this API, but it brings me to my next point which is shared clients.
2. Support for Shared Clients AND Time-shared Clients:
As I said before, we need the framework to be flexible enough to support
shared clients. The shared clients may not be applicable for all h/w
controllers, where a client has to send a series of operations that
needs to be sent to the remote before anyone else can use it. This is a
functional integration aspect, and is dictated by the nature of the
remote. For the time-shared clients, the remote must have some implicit
message protocol where in the remote is able to identify the macro
usecase through a specific message. The framework should allow the case
of shared clients, with the clients doing their own message demuxing.
You can take a look at my patches on top of the existing mailbox code
that allows both shared clients and atomic callbacks, and multi-instance
support. It is not that hard to incorporate similar concepts into this
code as well. It is based on properties published by the
controller/link. [I haven't added the tx callbacks and poll method, that
would actually mean I will be doing a very similar state machine to what
you have, no point in me spending time on that, unless we are gonna
absorb the current mailbox series.]
https://github.com/sumananna/mailbox/commits/mailbox-multi-atomic-dt-doc
3. Buffering/Size: I think we need to take a re-look at the whole tx
buffering mechanism. You are using an array that stores just the
pointers, which means that there are underlying assumptions that the
clients are keeping their buffer ptrs intact until the duration of the
transfer is done. This means that one cannot use local payload
variables for messages in the calling functions. I feel this is
unnecessary burden on the clients.
Secondly, we may not need the logic around client knows_txdone and
tx_buffering together. I understand the need for buffering when you have
TX_POLL or TX_IRQ, but as far as TX_ACK is concerned, it is a client
protocol-level knowledge, and so the tx buffering logic can remain with
the client itself. This should simplify the code a little bit and get
rid of the ipc_client_txdone API.
Looking at the current use-cases, I think OMAP might be the only one
which needs the buffering. The API that Loic added suggests that he
either sends a message and expects a response, or just sends a message
if the transport is free (I am not sure if he is using the API that uses
buffering currently). PL320 is the same as the first scenario that Loic has.
The other point is regarding the size field, I am not convinced that we
can take this out, and leave it between the client and the controller
implementation. I think the mailbox core should at least perform a check
based on the max size published by a controller driver. Like in the
PL320 patch conversion, the for loop for writing the data - how does it
know that the provided pointer is atleast 7 ints, and not smaller than
that?
Rest of the comments are in the code...
>
> diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> new file mode 100644
> index 0000000..c5ec93e
> --- /dev/null
> +++ b/drivers/mailbox/mailbox.c
> +
> +/*
> + * Called by a client to "put data on the h/w channel" so that if
> + * everything else is fine we don't need to do anything more locally
> + * for the remote to receive the data intact.
> + * In reality, the remote may receive it intact, corrupted or not at all.
> + * This could be called from atomic context as it simply
> + * queues the data and returns a token (request_token_t)
> + * against the request.
> + * The client is later notified of successful transmission of
> + * data over the channel via the 'txcb'. The client could in
> + * turn queue more messages from txcb.
> + */
> +request_token_t ipc_send_message(void *channel, void *data)
> +{
> + struct ipc_chan *chan = (struct ipc_chan *)channel;
> + request_token_t t;
> +
> + if (!chan)
> + return 0;
> +
> + t = _add_to_rbuf(chan, data);
> + if (!t)
> + printk("Try increasing MBOX_TX_QUEUE_LEN\n");
> +
> + _msg_submit(chan);
> +
> + if (chan->txdone_method == TXDONE_BY_POLL)
> + poll_txdone((unsigned long)chan->timer);
> +
> + if (chan->tx_block && chan->active_token) {
Assigning the tx_block at channel request time might not be flexible
enough. We should not impose that the client will request and release
channels with different modes if both scenarios are needed.
> + int ret;
> + init_completion(&chan->tx_complete);
> + ret = wait_for_completion_timeout(&chan->tx_complete,
> + chan->tx_tout);
> + if (ret == 0) {
> + t = 0;
> + tx_tick(chan, XFER_ERR);
> + }
> + }
> +
> + return t;
> +}
> +EXPORT_SYMBOL(ipc_send_message);
Do we need to add an API similar to mailbox_msg_send_receive_no_irq, or
are you leaving it to be client's responsibility?
> +
> +/*
> + * A client driver asks for exclusive use of a channel/mailbox.
> + * If assigned, the channel has to be 'freed' before it could
> + * be assigned to some other client.
> + * After assignment, any packet received on this channel will be
> + * handed over to the client via the 'rxcb' callback.
> + * The 'txcb' callback is used to notify client upon sending the
> + * packet over the channel, which may or may not have been yet
> + * read by the remote processor.
> + */
> +void *ipc_request_channel(struct ipc_client *cl)
> +{
> + struct ipc_chan *chan;
> + unsigned long flags;
> + int ret = 0;
> +
> + mutex_lock(&chpool_mutex);
> +
> + list_for_each_entry(chan, &ipc_channels, node)
I had a different design in mind here, maintain the list of controllers
globally instead of the channels, that way your search can be a bit
quicker.
> + if(!chan->assigned
> + && !strcmp(cl->chan_name, chan->chan_name)) {
> + spin_lock_irqsave(&chan->lock, flags);
> + chan->msg_free = 0;
> + chan->msg_count = 0;
> + chan->active_token = 0;
> + chan->rxcb = cl->rxcb;
> + chan->txcb = cl->txcb;
> + chan->assigned = true;
> + chan->tx_block = cl->tx_block;
> + if (!cl->tx_tout)
> + chan->tx_tout = ~0;
> + else
> + chan->tx_tout = msecs_to_jiffies(cl->tx_tout);
> + if (chan->txdone_method == TXDONE_BY_POLL
> + && cl->knows_txdone)
> + chan->txdone_method |= TXDONE_BY_ACK;
> + spin_unlock_irqrestore(&chan->lock, flags);
> + ret = 1;
> + break;
> + }
> +
> + mutex_unlock(&chpool_mutex);
> +
> + if (!ret) {
> + printk("Unable to assign mailbox(%s)\n", cl->chan_name);
> + return NULL;
> + }
> +
> + ret = chan->link_ops->startup(chan->link, cl->cntlr_data);
> + if (ret) {
> + printk("Unable to startup the link\n");
> + ipc_free_channel((void *)chan);
> + return NULL;
> + }
> +
> + return (void *)chan;
> +}
> +EXPORT_SYMBOL(ipc_request_channel);
> +
> +/*
> + * Call for IPC controller drivers to register a controller, adding
> + * its channels/mailboxes to the global pool.
> + */
> +int ipc_links_register(struct ipc_controller *ipc_con)
> +{
> + struct tx_poll_timer *timer = NULL;
> + struct ipc_chan *channel;
> + int i, num_links, txdone;
> +
> + /* Are you f***ing with us, sir? */
> + if (!ipc_con || !ipc_con->ops)
> + return -EINVAL;
> +
> + for (i = 0; ipc_con->links[i]; i++)
> + ;
> + if (!i)
> + return -EINVAL;
> + num_links = i;
> +
> + if (ipc_con->txdone_irq)
> + txdone = TXDONE_BY_IRQ;
> + else if (ipc_con->txdone_poll)
> + txdone = TXDONE_BY_POLL;
> + else /* It has to be at least ACK */
> + txdone = TXDONE_BY_ACK;
> +
> + if (txdone == TXDONE_BY_POLL) {
> + timer = kzalloc(sizeof(struct tx_poll_timer), GFP_KERNEL);
> + timer->period = ipc_con->txpoll_period;
> + timer->poll.function = &poll_txdone;
> + timer->poll.data = (unsigned long)timer;
> + init_timer(&timer->poll);
> + }
> +
> + channel = kzalloc(sizeof(struct ipc_chan) * num_links, GFP_KERNEL);
minor, but you might be aware of this already, failure check needed
> +
> + for (i = 0; i < num_links; i++) {
> + channel[i].timer = timer;
> + channel[i].assigned = false;
> + channel[i].txdone_method = txdone;
> + channel[i].link_ops = ipc_con->ops;
> + channel[i].link = ipc_con->links[i];
> + channel[i].link->api_priv = &channel[i];
> + snprintf(channel[i].chan_name, 32, "%s:%s",
> + ipc_con->controller_name,
> + ipc_con->links[i]->link_name);
> + spin_lock_init(&channel[i].lock);
> + BLOCKING_INIT_NOTIFIER_HEAD(&channel[i].avail);
> + mutex_lock(&chpool_mutex);
> + list_add_tail(&channel[i].node, &ipc_channels);
> + mutex_unlock(&chpool_mutex);
> + }
> +
> + return 0;
> +}
> +EXPORT_SYMBOL(ipc_links_register);
> +
> +static int __init ipc_mbox_init(void)
> +{
> + INIT_LIST_HEAD(&ipc_channels);
> + return 0;
> +}
> +subsys_initcall(ipc_mbox_init);
The ipc_mbox_init can be replaced by static list init of ipc_channels,
nothing much to be gained by defining this function. We expect the
mailbox.c to be built anyways whenever the mailbox framework is selected.
> diff --git a/include/linux/mailbox_client.h b/include/linux/mailbox_client.h
> new file mode 100644
> index 0000000..33545f6
> --- /dev/null
> +++ b/include/linux/mailbox_client.h
> +/**
> + * struct ipc_client - User of a mailbox
> + * @chan_name: the "controller:channel" this client wants
> + * @rxcb: atomic callback to provide client the data received
> + * @txcb: atomic callback to tell client of data transmission
> + * @tx_block: if the ipc_send_message should block until data is transmitted
> + * @tx_tout: Max block period in ms before TX is assumed failure
> + * @knows_txdone: if the client could run the TX state machine. Usually if
> + * the client receives some ACK packet for transmission. Unused if the
> + * controller already has TX_Done/RTR IRQ.
> + * @cntlr_data: Optional controller specific parameters during channel request
> + */
> +struct ipc_client {
> + char *chan_name;
> + void (*rxcb)(void *data);
> + void (*txcb)(request_token_t t, enum xfer_result r);
> + bool tx_block;
> + unsigned long tx_tout;
> + bool knows_txdone;
> + void *cntlr_data;
What is the current use-case for exposing cntrl_data through ipc_client?
I think this should be avoided and leave the controller configuration to
the controller driver implementation. This will be a problem for
multiple link scenarios.
> diff --git a/include/linux/mailbox_controller.h b/include/linux/mailbox_controller.h
> new file mode 100644
> index 0000000..defcfb3
> --- /dev/null
> +++ b/include/linux/mailbox_controller.h
> +
> +/**
> + * struct ipc_link - s/w representation of a communication link
> + * @link_name: Literal name assigned to the link. Physically
> + * identical channels may have the same name.
> + * @api_priv: hook for the API to map its private data on the link
> + * Controller driver must not touch it.
> + */
> +struct ipc_link {
> + char link_name[16];
> + void *api_priv;
> +};
this definitely needs a link-private void *ptr. PL320 doesn't have
multiple links in the controller, but this is a problem for OMAP & DBX500.
> +struct ipc_link_ops {
> + int (*send_data)(struct ipc_link *link, void *data);
> + int (*startup)(struct ipc_link *link, void *params);
> + void (*shutdown)(struct ipc_link *link);
> + bool (*last_tx_done)(struct ipc_link *link);
minor comment, maybe rename this to something that indicates link
busyness or readiness
> +};
> +
> +/**
> + * struct ipc_controller - Controller of a class of communication links
> + * @controller_name: Literal name of the controller.
> + * @ops: Operators that work on each communication link
> + * @links: Null terminated array of links.
> + * @txdone_irq: Indicates if the controller can report to API when the
> + * last transmitted data was read by the remote. Eg, if it has some
> + * TX ACK irq.
> + * @txdone_poll: If the controller can read but not report the TX done.
> + * Eg, is some register shows the TX status but no interrupt rises.
> + * Ignored if 'txdone_irq' is set.
> + * @txpoll_period: If 'txdone_poll' is in effect, the API polls for
> + * last TX's status after these many millisecs
> + */
> +struct ipc_controller {
> + char controller_name[16];
i think this can be avoided and use the underlying dev-name directly
based on the controller device. Adding a struct device pointer would
automatically allow you to use the name from the probe.
> + struct ipc_link_ops *ops;
I think this should be a property of the individual link for
flexibility. Right now, it probably doesn't make a difference (as seen
from the current mailbox code), but the moment we have one link behaving
differently this will make the ops implementation a bit messy.
I think we also need a controller-specific ops, to put common stuff
between multiple links within the controller.
> + struct ipc_link **links;
> + bool txdone_irq;
> + bool txdone_poll;
> + unsigned txpoll_period;
> +};
> +
^ permalink raw reply [flat|nested] 90+ messages in thread* Re: [RFC 2/3] mailbox: Introduce a new common API
@ 2013-05-04 2:20 ` Suman Anna
0 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-05-04 2:20 UTC (permalink / raw)
To: jassisinghbrar
Cc: loic.pallardy, arnd, linux-kernel, linux-arm-kernel, andy.green,
Jassi Brar
Hi Jassi,
On 04/27/2013 01:14 PM, jassisinghbrar@gmail.com wrote:
> From: Jassi Brar <jaswinder.singh@linaro.org>
>
> Introduce common framework for client/protocol drivers and
> controller drivers of Inter-Processor-Communication (IPC).
>
> Client driver developers should have a look at
> include/linux/mailbox_client.h to understand the part of
> the API exposed to client drivers.
> Similarly controller driver developers should have a look
> at include/linux/mailbox_controller.h
This implementation looks decent based on your design points. These
are the open/contention points that needs to be sorted out.
I think understanding the OMAP mailbox architecture also helps you,
since this series currently addressed PL320, but we will run into
some issues when adopting for OMAP as is. OMAP has a single mailbox IP,
each with multiple h/w FIFOs (fifo of length 4, and message size of a
u32). Each IP can cause one or more irq (usually 1) into the linux host.
It has status registers for number of messages in a FIFO (Rx), FIFO full
(Tx non-readiness/busy state), and interrupts for both Rx and Tx. There
are registers indicating the source of the interrupt, and these are per
FIFO. The Tx interrupt source is really for indicating Tx readiness or
that a fifo has open room for sending messages. This will keep on firing
as long as the FIFO is not-full, so we usually configure this only when
the FIFO is full and disable it the moment it is fired. It is re-enabled
when the h/w FIFO is full again, and we use s/w buffering in between.
The Rx interrupt is fired as long as there is a message in any FIFO
which has been configured to interrupt the host, so we have to empty all
the messages for that interrupt source.
Anyway, here is a summary of the open points that we have:
1. Atomic Callbacks:
The current code provides some sort of buffering on Tx, but imposes the
restriction that the clients do the buffering on Rx. This is main
concern for OMAP. We have multiple links on a single interrupt and
multiple messages per link on a interrupt source. Pushing this logic
into a client (remoteproc for us) is not right, that code does not
belong to the remoteproc driver, and neither is adding another layer in
between. It also kills the shared clients idea. The current
ipc_link_received_data seems open enough for me to use by calling it
from a work-queue, and have the OMAP controller driver take care of Rx
buffering (will test this next week). Restricting ipc_link_received_data
to atomic callbacks might have been enough for single transport
payloads, but we do have a FIFO. I could possibly make it work using
this API, but it brings me to my next point which is shared clients.
2. Support for Shared Clients AND Time-shared Clients:
As I said before, we need the framework to be flexible enough to support
shared clients. The shared clients may not be applicable for all h/w
controllers, where a client has to send a series of operations that
needs to be sent to the remote before anyone else can use it. This is a
functional integration aspect, and is dictated by the nature of the
remote. For the time-shared clients, the remote must have some implicit
message protocol where in the remote is able to identify the macro
usecase through a specific message. The framework should allow the case
of shared clients, with the clients doing their own message demuxing.
You can take a look at my patches on top of the existing mailbox code
that allows both shared clients and atomic callbacks, and multi-instance
support. It is not that hard to incorporate similar concepts into this
code as well. It is based on properties published by the
controller/link. [I haven't added the tx callbacks and poll method, that
would actually mean I will be doing a very similar state machine to what
you have, no point in me spending time on that, unless we are gonna
absorb the current mailbox series.]
https://github.com/sumananna/mailbox/commits/mailbox-multi-atomic-dt-doc
3. Buffering/Size: I think we need to take a re-look at the whole tx
buffering mechanism. You are using an array that stores just the
pointers, which means that there are underlying assumptions that the
clients are keeping their buffer ptrs intact until the duration of the
transfer is done. This means that one cannot use local payload
variables for messages in the calling functions. I feel this is
unnecessary burden on the clients.
Secondly, we may not need the logic around client knows_txdone and
tx_buffering together. I understand the need for buffering when you have
TX_POLL or TX_IRQ, but as far as TX_ACK is concerned, it is a client
protocol-level knowledge, and so the tx buffering logic can remain with
the client itself. This should simplify the code a little bit and get
rid of the ipc_client_txdone API.
Looking at the current use-cases, I think OMAP might be the only one
which needs the buffering. The API that Loic added suggests that he
either sends a message and expects a response, or just sends a message
if the transport is free (I am not sure if he is using the API that uses
buffering currently). PL320 is the same as the first scenario that Loic has.
The other point is regarding the size field, I am not convinced that we
can take this out, and leave it between the client and the controller
implementation. I think the mailbox core should at least perform a check
based on the max size published by a controller driver. Like in the
PL320 patch conversion, the for loop for writing the data - how does it
know that the provided pointer is atleast 7 ints, and not smaller than
that?
Rest of the comments are in the code...
>
> diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
> new file mode 100644
> index 0000000..c5ec93e
> --- /dev/null
> +++ b/drivers/mailbox/mailbox.c
> +
> +/*
> + * Called by a client to "put data on the h/w channel" so that if
> + * everything else is fine we don't need to do anything more locally
> + * for the remote to receive the data intact.
> + * In reality, the remote may receive it intact, corrupted or not at all.
> + * This could be called from atomic context as it simply
> + * queues the data and returns a token (request_token_t)
> + * against the request.
> + * The client is later notified of successful transmission of
> + * data over the channel via the 'txcb'. The client could in
> + * turn queue more messages from txcb.
> + */
> +request_token_t ipc_send_message(void *channel, void *data)
> +{
> + struct ipc_chan *chan = (struct ipc_chan *)channel;
> + request_token_t t;
> +
> + if (!chan)
> + return 0;
> +
> + t = _add_to_rbuf(chan, data);
> + if (!t)
> + printk("Try increasing MBOX_TX_QUEUE_LEN\n");
> +
> + _msg_submit(chan);
> +
> + if (chan->txdone_method == TXDONE_BY_POLL)
> + poll_txdone((unsigned long)chan->timer);
> +
> + if (chan->tx_block && chan->active_token) {
Assigning the tx_block at channel request time might not be flexible
enough. We should not impose that the client will request and release
channels with different modes if both scenarios are needed.
> + int ret;
> + init_completion(&chan->tx_complete);
> + ret = wait_for_completion_timeout(&chan->tx_complete,
> + chan->tx_tout);
> + if (ret == 0) {
> + t = 0;
> + tx_tick(chan, XFER_ERR);
> + }
> + }
> +
> + return t;
> +}
> +EXPORT_SYMBOL(ipc_send_message);
Do we need to add an API similar to mailbox_msg_send_receive_no_irq, or
are you leaving it to be client's responsibility?
> +
> +/*
> + * A client driver asks for exclusive use of a channel/mailbox.
> + * If assigned, the channel has to be 'freed' before it could
> + * be assigned to some other client.
> + * After assignment, any packet received on this channel will be
> + * handed over to the client via the 'rxcb' callback.
> + * The 'txcb' callback is used to notify client upon sending the
> + * packet over the channel, which may or may not have been yet
> + * read by the remote processor.
> + */
> +void *ipc_request_channel(struct ipc_client *cl)
> +{
> + struct ipc_chan *chan;
> + unsigned long flags;
> + int ret = 0;
> +
> + mutex_lock(&chpool_mutex);
> +
> + list_for_each_entry(chan, &ipc_channels, node)
I had a different design in mind here, maintain the list of controllers
globally instead of the channels, that way your search can be a bit
quicker.
> + if(!chan->assigned
> + && !strcmp(cl->chan_name, chan->chan_name)) {
> + spin_lock_irqsave(&chan->lock, flags);
> + chan->msg_free = 0;
> + chan->msg_count = 0;
> + chan->active_token = 0;
> + chan->rxcb = cl->rxcb;
> + chan->txcb = cl->txcb;
> + chan->assigned = true;
> + chan->tx_block = cl->tx_block;
> + if (!cl->tx_tout)
> + chan->tx_tout = ~0;
> + else
> + chan->tx_tout = msecs_to_jiffies(cl->tx_tout);
> + if (chan->txdone_method == TXDONE_BY_POLL
> + && cl->knows_txdone)
> + chan->txdone_method |= TXDONE_BY_ACK;
> + spin_unlock_irqrestore(&chan->lock, flags);
> + ret = 1;
> + break;
> + }
> +
> + mutex_unlock(&chpool_mutex);
> +
> + if (!ret) {
> + printk("Unable to assign mailbox(%s)\n", cl->chan_name);
> + return NULL;
> + }
> +
> + ret = chan->link_ops->startup(chan->link, cl->cntlr_data);
> + if (ret) {
> + printk("Unable to startup the link\n");
> + ipc_free_channel((void *)chan);
> + return NULL;
> + }
> +
> + return (void *)chan;
> +}
> +EXPORT_SYMBOL(ipc_request_channel);
> +
> +/*
> + * Call for IPC controller drivers to register a controller, adding
> + * its channels/mailboxes to the global pool.
> + */
> +int ipc_links_register(struct ipc_controller *ipc_con)
> +{
> + struct tx_poll_timer *timer = NULL;
> + struct ipc_chan *channel;
> + int i, num_links, txdone;
> +
> + /* Are you f***ing with us, sir? */
> + if (!ipc_con || !ipc_con->ops)
> + return -EINVAL;
> +
> + for (i = 0; ipc_con->links[i]; i++)
> + ;
> + if (!i)
> + return -EINVAL;
> + num_links = i;
> +
> + if (ipc_con->txdone_irq)
> + txdone = TXDONE_BY_IRQ;
> + else if (ipc_con->txdone_poll)
> + txdone = TXDONE_BY_POLL;
> + else /* It has to be at least ACK */
> + txdone = TXDONE_BY_ACK;
> +
> + if (txdone == TXDONE_BY_POLL) {
> + timer = kzalloc(sizeof(struct tx_poll_timer), GFP_KERNEL);
> + timer->period = ipc_con->txpoll_period;
> + timer->poll.function = &poll_txdone;
> + timer->poll.data = (unsigned long)timer;
> + init_timer(&timer->poll);
> + }
> +
> + channel = kzalloc(sizeof(struct ipc_chan) * num_links, GFP_KERNEL);
minor, but you might be aware of this already, failure check needed
> +
> + for (i = 0; i < num_links; i++) {
> + channel[i].timer = timer;
> + channel[i].assigned = false;
> + channel[i].txdone_method = txdone;
> + channel[i].link_ops = ipc_con->ops;
> + channel[i].link = ipc_con->links[i];
> + channel[i].link->api_priv = &channel[i];
> + snprintf(channel[i].chan_name, 32, "%s:%s",
> + ipc_con->controller_name,
> + ipc_con->links[i]->link_name);
> + spin_lock_init(&channel[i].lock);
> + BLOCKING_INIT_NOTIFIER_HEAD(&channel[i].avail);
> + mutex_lock(&chpool_mutex);
> + list_add_tail(&channel[i].node, &ipc_channels);
> + mutex_unlock(&chpool_mutex);
> + }
> +
> + return 0;
> +}
> +EXPORT_SYMBOL(ipc_links_register);
> +
> +static int __init ipc_mbox_init(void)
> +{
> + INIT_LIST_HEAD(&ipc_channels);
> + return 0;
> +}
> +subsys_initcall(ipc_mbox_init);
The ipc_mbox_init can be replaced by static list init of ipc_channels,
nothing much to be gained by defining this function. We expect the
mailbox.c to be built anyways whenever the mailbox framework is selected.
> diff --git a/include/linux/mailbox_client.h b/include/linux/mailbox_client.h
> new file mode 100644
> index 0000000..33545f6
> --- /dev/null
> +++ b/include/linux/mailbox_client.h
> +/**
> + * struct ipc_client - User of a mailbox
> + * @chan_name: the "controller:channel" this client wants
> + * @rxcb: atomic callback to provide client the data received
> + * @txcb: atomic callback to tell client of data transmission
> + * @tx_block: if the ipc_send_message should block until data is transmitted
> + * @tx_tout: Max block period in ms before TX is assumed failure
> + * @knows_txdone: if the client could run the TX state machine. Usually if
> + * the client receives some ACK packet for transmission. Unused if the
> + * controller already has TX_Done/RTR IRQ.
> + * @cntlr_data: Optional controller specific parameters during channel request
> + */
> +struct ipc_client {
> + char *chan_name;
> + void (*rxcb)(void *data);
> + void (*txcb)(request_token_t t, enum xfer_result r);
> + bool tx_block;
> + unsigned long tx_tout;
> + bool knows_txdone;
> + void *cntlr_data;
What is the current use-case for exposing cntrl_data through ipc_client?
I think this should be avoided and leave the controller configuration to
the controller driver implementation. This will be a problem for
multiple link scenarios.
> diff --git a/include/linux/mailbox_controller.h b/include/linux/mailbox_controller.h
> new file mode 100644
> index 0000000..defcfb3
> --- /dev/null
> +++ b/include/linux/mailbox_controller.h
> +
> +/**
> + * struct ipc_link - s/w representation of a communication link
> + * @link_name: Literal name assigned to the link. Physically
> + * identical channels may have the same name.
> + * @api_priv: hook for the API to map its private data on the link
> + * Controller driver must not touch it.
> + */
> +struct ipc_link {
> + char link_name[16];
> + void *api_priv;
> +};
this definitely needs a link-private void *ptr. PL320 doesn't have
multiple links in the controller, but this is a problem for OMAP & DBX500.
> +struct ipc_link_ops {
> + int (*send_data)(struct ipc_link *link, void *data);
> + int (*startup)(struct ipc_link *link, void *params);
> + void (*shutdown)(struct ipc_link *link);
> + bool (*last_tx_done)(struct ipc_link *link);
minor comment, maybe rename this to something that indicates link
busyness or readiness
> +};
> +
> +/**
> + * struct ipc_controller - Controller of a class of communication links
> + * @controller_name: Literal name of the controller.
> + * @ops: Operators that work on each communication link
> + * @links: Null terminated array of links.
> + * @txdone_irq: Indicates if the controller can report to API when the
> + * last transmitted data was read by the remote. Eg, if it has some
> + * TX ACK irq.
> + * @txdone_poll: If the controller can read but not report the TX done.
> + * Eg, is some register shows the TX status but no interrupt rises.
> + * Ignored if 'txdone_irq' is set.
> + * @txpoll_period: If 'txdone_poll' is in effect, the API polls for
> + * last TX's status after these many millisecs
> + */
> +struct ipc_controller {
> + char controller_name[16];
i think this can be avoided and use the underlying dev-name directly
based on the controller device. Adding a struct device pointer would
automatically allow you to use the name from the probe.
> + struct ipc_link_ops *ops;
I think this should be a property of the individual link for
flexibility. Right now, it probably doesn't make a difference (as seen
from the current mailbox code), but the moment we have one link behaving
differently this will make the ops implementation a bit messy.
I think we also need a controller-specific ops, to put common stuff
between multiple links within the controller.
> + struct ipc_link **links;
> + bool txdone_irq;
> + bool txdone_poll;
> + unsigned txpoll_period;
> +};
> +
^ permalink raw reply [flat|nested] 90+ messages in thread* [RFC 2/3] mailbox: Introduce a new common API
2013-05-04 2:20 ` Suman Anna
@ 2013-05-04 19:08 ` Jassi Brar
-1 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-05-04 19:08 UTC (permalink / raw)
To: linux-arm-kernel
Hi Suman,
On 4 May 2013 07:50, Suman Anna <s-anna@ti.com> wrote:
> Hi Jassi,
>
> On 04/27/2013 01:14 PM, jassisinghbrar at gmail.com wrote:
>> From: Jassi Brar <jaswinder.singh@linaro.org>
>>
>> Introduce common framework for client/protocol drivers and
>> controller drivers of Inter-Processor-Communication (IPC).
>>
>> Client driver developers should have a look at
>> include/linux/mailbox_client.h to understand the part of
>> the API exposed to client drivers.
>> Similarly controller driver developers should have a look
>> at include/linux/mailbox_controller.h
>
> This implementation looks decent based on your design points. These
> are the open/contention points that needs to be sorted out.
>
> I think understanding the OMAP mailbox architecture also helps you,
> since this series currently addressed PL320, but we will run into
> some issues when adopting for OMAP as is. OMAP has a single mailbox IP,
> each with multiple h/w FIFOs (fifo of length 4, and message size of a
> u32). Each IP can cause one or more irq (usually 1) into the linux host.
> It has status registers for number of messages in a FIFO (Rx), FIFO full
> (Tx non-readiness/busy state), and interrupts for both Rx and Tx. There
> are registers indicating the source of the interrupt, and these are per
> FIFO. The Tx interrupt source is really for indicating Tx readiness or
> that a fifo has open room for sending messages. This will keep on firing
> as long as the FIFO is not-full, so we usually configure this only when
> the FIFO is full and disable it the moment it is fired. It is re-enabled
> when the h/w FIFO is full again, and we use s/w buffering in between.
> The Rx interrupt is fired as long as there is a message in any FIFO
> which has been configured to interrupt the host, so we have to empty all
> the messages for that interrupt source.
>
Yeah, thanks for explanation. I've worked on OMAP for almost 2 yrs
now, though not mailbox. I did have a look at the OMAP4430 trm.
I understand OMAP's MBox doesn't really have TX-Done/RTR interrupt,
it only has "Tx Buffer Not Full" interrupt which serves the purpose
only for the window when there are at least 4 TX messages pending. Now
the driver could switch between Polling and IRQ at runtime depending
upon the buffer filled extent, OR simply work in polling mode all the
time. IMHO the controller driver should opt for the latter. Though it
might be interesting to profile out of total transfers during a
period, how many are actually queued till depth of 4.
> Anyway, here is a summary of the open points that we have:
> 1. Atomic Callbacks:
> The current code provides some sort of buffering on Tx, but imposes the
> restriction that the clients do the buffering on Rx. This is main
> concern for OMAP.
I am afraid a common API can't do without buffering TX and it can't do
by buffering RX.
The client(s) can always generate TX requests at a rate greater than
the API could transmit on the physical link. So as much as we dislike
it, we have to buffer TX requests, otherwise N clients would.
OTOH Buffering received packets in the API doesn't help anybody, it
only incurs unavoidable latencies on clients. Only clients, that need
to take non-atomic actions upon RX, would need to buffer RX. Other
clients should not suffer.
IMHO if clients on OMAP need to buffer RX, let us keep it OMAP
specific. If number of such platforms rise in future we could move
that as an _optional_ helper API on top, that does RX buffering on
behalf of clients ?
> 2. Support for Shared Clients AND Time-shared Clients:
> As I said before, we need the framework to be flexible enough to support
> shared clients. The shared clients may not be applicable for all h/w
> controllers, where a client has to send a series of operations that
> needs to be sent to the remote before anyone else can use it. This is a
> functional integration aspect, and is dictated by the nature of the
> remote. For the time-shared clients, the remote must have some implicit
> message protocol where in the remote is able to identify the macro
> usecase through a specific message. The framework should allow the case
> of shared clients, with the clients doing their own message demuxing.
>
If the API provides shared ownership of a mailbox, it won't work for
clients that need exclusive ownership (like 'server' side
implementation of a protocol).
OTOH if the API provides exclusive ownership, it is still possible to
emulate shared ownership by simply publishing the mailbox handle (void
*chan) globally. It works for all.
>
> 3. Buffering/Size: I think we need to take a re-look at the whole tx
> buffering mechanism. You are using an array that stores just the
> pointers, which means that there are underlying assumptions that the
> clients are keeping their buffer ptrs intact until the duration of the
> transfer is done. This means that one cannot use local payload
> variables for messages in the calling functions. I feel this is
> unnecessary burden on the clients.
>
Most of the clients won't queue more than 1 request at a time. And
then, isn't it only natural that clients don't mess with requests
after submitting them ? I see mailbox clients working quite like I2C
clients.
> Secondly, we may not need the logic around client knows_txdone and
> tx_buffering together. I understand the need for buffering when you have
> TX_POLL or TX_IRQ, but as far as TX_ACK is concerned, it is a client
> protocol-level knowledge, and so the tx buffering logic can remain with
> the client itself. This should simplify the code a little bit and get
> rid of the ipc_client_txdone API.
>
Yeah I had it that way originally. But then I realized we could be
running an 'ACK Packet' protocol over a controller that supports only
POLL. In that case the controller's poll doesn't override client's
knows_txdone because we want the client to cut-short the next poll
delay as soon as it gets an ACK packet.
> Looking at the current use-cases, I think OMAP might be the only one
> which needs the buffering. The API that Loic added suggests that he
> either sends a message and expects a response, or just sends a message
> if the transport is free (I am not sure if he is using the API that uses
> buffering currently). PL320 is the same as the first scenario that Loic has.
>
Yeah I too expect TX buffering to be very rare.
> The other point is regarding the size field, I am not convinced that we
> can take this out, and leave it between the client and the controller
> implementation. I think the mailbox core should at least perform a check
> based on the max size published by a controller driver. Like in the
> PL320 patch conversion, the for loop for writing the data - how does it
> know that the provided pointer is atleast 7 ints, and not smaller than
> that?
>
First of all, PL320 is a highly configurable and programmable IP.
pl320.c is very specific to Highbank's usage, it is not usable as such
on other platforms. There are many things hardcoded and tacitly
assumed in the driver. I just changed the API, didn't add anything
new.
Secondly, I believe we acknowledged the fact that a client can't do
without controller specific knowledge. So in proper implementation the
controller's header would look like
struct foobar_con_mssg {
int datalen; /* size of received packet in bytes */
u8 *buf; /* buffer that holds the packet data */
.....
};
So instead of telling the API the 'datalen' and the remaining
structure of each message, we simply pass "struct foobar_con_mssg *"
as a void* to the API.
>> +
>> +/*
>> + * Called by a client to "put data on the h/w channel" so that if
>> + * everything else is fine we don't need to do anything more locally
>> + * for the remote to receive the data intact.
>> + * In reality, the remote may receive it intact, corrupted or not at all.
>> + * This could be called from atomic context as it simply
>> + * queues the data and returns a token (request_token_t)
>> + * against the request.
>> + * The client is later notified of successful transmission of
>> + * data over the channel via the 'txcb'. The client could in
>> + * turn queue more messages from txcb.
>> + */
>> +request_token_t ipc_send_message(void *channel, void *data)
>> +{
>> + struct ipc_chan *chan = (struct ipc_chan *)channel;
>> + request_token_t t;
>> +
>> + if (!chan)
>> + return 0;
>> +
>> + t = _add_to_rbuf(chan, data);
>> + if (!t)
>> + printk("Try increasing MBOX_TX_QUEUE_LEN\n");
>> +
>> + _msg_submit(chan);
>> +
>> + if (chan->txdone_method == TXDONE_BY_POLL)
>> + poll_txdone((unsigned long)chan->timer);
>> +
>> + if (chan->tx_block && chan->active_token) {
>
> Assigning the tx_block at channel request time might not be flexible
> enough. We should not impose that the client will request and release
> channels with different modes if both scenarios are needed.
>
If there is one blocking request, no more could arrive (blocking or
non-blocking). The increased code complexity doesn't justify the
rarely used feature. I don't see many use-cases when the client can't
release-request the channel. It's made like opening a file - blocking
vs non-blocking is the mode you open the resource.
>> + int ret;
>> + init_completion(&chan->tx_complete);
>> + ret = wait_for_completion_timeout(&chan->tx_complete,
>> + chan->tx_tout);
>> + if (ret == 0) {
>> + t = 0;
>> + tx_tick(chan, XFER_ERR);
>> + }
>> + }
>> +
>> + return t;
>> +}
>> +EXPORT_SYMBOL(ipc_send_message);
>
> Do we need to add an API similar to mailbox_msg_send_receive_no_irq, or
> are you leaving it to be client's responsibility?
>
Yes the API provides enough ways for a client to do that.
>> +
>> +/*
>> + * A client driver asks for exclusive use of a channel/mailbox.
>> + * If assigned, the channel has to be 'freed' before it could
>> + * be assigned to some other client.
>> + * After assignment, any packet received on this channel will be
>> + * handed over to the client via the 'rxcb' callback.
>> + * The 'txcb' callback is used to notify client upon sending the
>> + * packet over the channel, which may or may not have been yet
>> + * read by the remote processor.
>> + */
>> +void *ipc_request_channel(struct ipc_client *cl)
>> +{
>> + struct ipc_chan *chan;
>> + unsigned long flags;
>> + int ret = 0;
>> +
>> + mutex_lock(&chpool_mutex);
>> +
>> + list_for_each_entry(chan, &ipc_channels, node)
> I had a different design in mind here, maintain the list of controllers
> globally instead of the channels, that way your search can be a bit
> quicker.
>
Yeah, I changed that to it last minute :)
The API has no real use of controller nodes, it works solely on a
global pool of mailboxes. I tried to reflect that in code.
>> +
>> + if (txdone == TXDONE_BY_POLL) {
>> + timer = kzalloc(sizeof(struct tx_poll_timer), GFP_KERNEL);
>> + timer->period = ipc_con->txpoll_period;
>> + timer->poll.function = &poll_txdone;
>> + timer->poll.data = (unsigned long)timer;
>> + init_timer(&timer->poll);
>> + }
>> +
>> + channel = kzalloc(sizeof(struct ipc_chan) * num_links, GFP_KERNEL);
>
> minor, but you might be aware of this already, failure check needed
>
No, I overlooked. Thanks.
>> +
>> +static int __init ipc_mbox_init(void)
>> +{
>> + INIT_LIST_HEAD(&ipc_channels);
>> + return 0;
>> +}
>> +subsys_initcall(ipc_mbox_init);
>
> The ipc_mbox_init can be replaced by static list init of ipc_channels,
> nothing much to be gained by defining this function. We expect the
> mailbox.c to be built anyways whenever the mailbox framework is selected.
>
Yes, you are right. Thanks.
>> + * @cntlr_data: Optional controller specific parameters during channel request
>> + */
>> +struct ipc_client {
>> + char *chan_name;
>> + void (*rxcb)(void *data);
>> + void (*txcb)(request_token_t t, enum xfer_result r);
>> + bool tx_block;
>> + unsigned long tx_tout;
>> + bool knows_txdone;
>> + void *cntlr_data;
>
> What is the current use-case for exposing cntrl_data through ipc_client?
> I think this should be avoided and leave the controller configuration to
> the controller driver implementation. This will be a problem for
> multiple link scenarios.
>
There are some highly programmable controllers that, if the driver is
proper, could change behavior runtime. For ex, if the controller
supports, a client might want to broadcast messages to multiple
remotes or listen for messages from more than one remote source or
both. The client could, say, specify bitmasks for such source/sink
remotes via cntlr_data while requesting the mailbox. PL320 could work
in such mode if rest all fits.
>> +struct ipc_link {
>> + char link_name[16];
>> + void *api_priv;
>> +};
>
> this definitely needs a link-private void *ptr. PL320 doesn't have
> multiple links in the controller, but this is a problem for OMAP & DBX500.
>
The API keeps the ipc_link provided by the controller driver, which
could have it embedded in its private representation of the h/w links.
For ex, my controller has 6 links, and its structure contains 6
element array of 'struct myhw_link' which embeds an ipc_link. And I
use container_of().
>> +struct ipc_link_ops {
>> + int (*send_data)(struct ipc_link *link, void *data);
>> + int (*startup)(struct ipc_link *link, void *params);
>> + void (*shutdown)(struct ipc_link *link);
>> + bool (*last_tx_done)(struct ipc_link *link);
> minor comment, maybe rename this to something that indicates link
> busyness or readiness
>
The API gives the controller only one TX at a time, so I chose the
name. What do you suggest?
>> +};
>> +
>> +/**
>> + * struct ipc_controller - Controller of a class of communication links
>> + * @controller_name: Literal name of the controller.
>> + * @ops: Operators that work on each communication link
>> + * @links: Null terminated array of links.
>> + * @txdone_irq: Indicates if the controller can report to API when the
>> + * last transmitted data was read by the remote. Eg, if it has some
>> + * TX ACK irq.
>> + * @txdone_poll: If the controller can read but not report the TX done.
>> + * Eg, is some register shows the TX status but no interrupt rises.
>> + * Ignored if 'txdone_irq' is set.
>> + * @txpoll_period: If 'txdone_poll' is in effect, the API polls for
>> + * last TX's status after these many millisecs
>> + */
>> +struct ipc_controller {
>> + char controller_name[16];
> i think this can be avoided and use the underlying dev-name directly
> based on the controller device. Adding a struct device pointer would
> automatically allow you to use the name from the probe.
>
Ah ha... yes. Thanks.
>> + struct ipc_link_ops *ops;
> I think this should be a property of the individual link for
> flexibility. Right now, it probably doesn't make a difference (as seen
> from the current mailbox code), but the moment we have one link behaving
> differently this will make the ops implementation a bit messy.
>
> I think we also need a controller-specific ops, to put common stuff
> between multiple links within the controller.
>
A controller is assumed to be a bunch of identical links/mailboxes.
However the cntlr_data could still help the client specify mode in
which it wants a mailbox behave.
Thanks and Regards.
-Jassi
^ permalink raw reply [flat|nested] 90+ messages in thread* Re: [RFC 2/3] mailbox: Introduce a new common API
@ 2013-05-04 19:08 ` Jassi Brar
0 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-05-04 19:08 UTC (permalink / raw)
To: Suman Anna
Cc: Jassi Brar, Loic PALLARDY (loic.pallardy@st.com), Arnd Bergmann,
lkml, linux-arm-kernel@lists.infradead.org, Andy Green
Hi Suman,
On 4 May 2013 07:50, Suman Anna <s-anna@ti.com> wrote:
> Hi Jassi,
>
> On 04/27/2013 01:14 PM, jassisinghbrar@gmail.com wrote:
>> From: Jassi Brar <jaswinder.singh@linaro.org>
>>
>> Introduce common framework for client/protocol drivers and
>> controller drivers of Inter-Processor-Communication (IPC).
>>
>> Client driver developers should have a look at
>> include/linux/mailbox_client.h to understand the part of
>> the API exposed to client drivers.
>> Similarly controller driver developers should have a look
>> at include/linux/mailbox_controller.h
>
> This implementation looks decent based on your design points. These
> are the open/contention points that needs to be sorted out.
>
> I think understanding the OMAP mailbox architecture also helps you,
> since this series currently addressed PL320, but we will run into
> some issues when adopting for OMAP as is. OMAP has a single mailbox IP,
> each with multiple h/w FIFOs (fifo of length 4, and message size of a
> u32). Each IP can cause one or more irq (usually 1) into the linux host.
> It has status registers for number of messages in a FIFO (Rx), FIFO full
> (Tx non-readiness/busy state), and interrupts for both Rx and Tx. There
> are registers indicating the source of the interrupt, and these are per
> FIFO. The Tx interrupt source is really for indicating Tx readiness or
> that a fifo has open room for sending messages. This will keep on firing
> as long as the FIFO is not-full, so we usually configure this only when
> the FIFO is full and disable it the moment it is fired. It is re-enabled
> when the h/w FIFO is full again, and we use s/w buffering in between.
> The Rx interrupt is fired as long as there is a message in any FIFO
> which has been configured to interrupt the host, so we have to empty all
> the messages for that interrupt source.
>
Yeah, thanks for explanation. I've worked on OMAP for almost 2 yrs
now, though not mailbox. I did have a look at the OMAP4430 trm.
I understand OMAP's MBox doesn't really have TX-Done/RTR interrupt,
it only has "Tx Buffer Not Full" interrupt which serves the purpose
only for the window when there are at least 4 TX messages pending. Now
the driver could switch between Polling and IRQ at runtime depending
upon the buffer filled extent, OR simply work in polling mode all the
time. IMHO the controller driver should opt for the latter. Though it
might be interesting to profile out of total transfers during a
period, how many are actually queued till depth of 4.
> Anyway, here is a summary of the open points that we have:
> 1. Atomic Callbacks:
> The current code provides some sort of buffering on Tx, but imposes the
> restriction that the clients do the buffering on Rx. This is main
> concern for OMAP.
I am afraid a common API can't do without buffering TX and it can't do
by buffering RX.
The client(s) can always generate TX requests at a rate greater than
the API could transmit on the physical link. So as much as we dislike
it, we have to buffer TX requests, otherwise N clients would.
OTOH Buffering received packets in the API doesn't help anybody, it
only incurs unavoidable latencies on clients. Only clients, that need
to take non-atomic actions upon RX, would need to buffer RX. Other
clients should not suffer.
IMHO if clients on OMAP need to buffer RX, let us keep it OMAP
specific. If number of such platforms rise in future we could move
that as an _optional_ helper API on top, that does RX buffering on
behalf of clients ?
> 2. Support for Shared Clients AND Time-shared Clients:
> As I said before, we need the framework to be flexible enough to support
> shared clients. The shared clients may not be applicable for all h/w
> controllers, where a client has to send a series of operations that
> needs to be sent to the remote before anyone else can use it. This is a
> functional integration aspect, and is dictated by the nature of the
> remote. For the time-shared clients, the remote must have some implicit
> message protocol where in the remote is able to identify the macro
> usecase through a specific message. The framework should allow the case
> of shared clients, with the clients doing their own message demuxing.
>
If the API provides shared ownership of a mailbox, it won't work for
clients that need exclusive ownership (like 'server' side
implementation of a protocol).
OTOH if the API provides exclusive ownership, it is still possible to
emulate shared ownership by simply publishing the mailbox handle (void
*chan) globally. It works for all.
>
> 3. Buffering/Size: I think we need to take a re-look at the whole tx
> buffering mechanism. You are using an array that stores just the
> pointers, which means that there are underlying assumptions that the
> clients are keeping their buffer ptrs intact until the duration of the
> transfer is done. This means that one cannot use local payload
> variables for messages in the calling functions. I feel this is
> unnecessary burden on the clients.
>
Most of the clients won't queue more than 1 request at a time. And
then, isn't it only natural that clients don't mess with requests
after submitting them ? I see mailbox clients working quite like I2C
clients.
> Secondly, we may not need the logic around client knows_txdone and
> tx_buffering together. I understand the need for buffering when you have
> TX_POLL or TX_IRQ, but as far as TX_ACK is concerned, it is a client
> protocol-level knowledge, and so the tx buffering logic can remain with
> the client itself. This should simplify the code a little bit and get
> rid of the ipc_client_txdone API.
>
Yeah I had it that way originally. But then I realized we could be
running an 'ACK Packet' protocol over a controller that supports only
POLL. In that case the controller's poll doesn't override client's
knows_txdone because we want the client to cut-short the next poll
delay as soon as it gets an ACK packet.
> Looking at the current use-cases, I think OMAP might be the only one
> which needs the buffering. The API that Loic added suggests that he
> either sends a message and expects a response, or just sends a message
> if the transport is free (I am not sure if he is using the API that uses
> buffering currently). PL320 is the same as the first scenario that Loic has.
>
Yeah I too expect TX buffering to be very rare.
> The other point is regarding the size field, I am not convinced that we
> can take this out, and leave it between the client and the controller
> implementation. I think the mailbox core should at least perform a check
> based on the max size published by a controller driver. Like in the
> PL320 patch conversion, the for loop for writing the data - how does it
> know that the provided pointer is atleast 7 ints, and not smaller than
> that?
>
First of all, PL320 is a highly configurable and programmable IP.
pl320.c is very specific to Highbank's usage, it is not usable as such
on other platforms. There are many things hardcoded and tacitly
assumed in the driver. I just changed the API, didn't add anything
new.
Secondly, I believe we acknowledged the fact that a client can't do
without controller specific knowledge. So in proper implementation the
controller's header would look like
struct foobar_con_mssg {
int datalen; /* size of received packet in bytes */
u8 *buf; /* buffer that holds the packet data */
.....
};
So instead of telling the API the 'datalen' and the remaining
structure of each message, we simply pass "struct foobar_con_mssg *"
as a void* to the API.
>> +
>> +/*
>> + * Called by a client to "put data on the h/w channel" so that if
>> + * everything else is fine we don't need to do anything more locally
>> + * for the remote to receive the data intact.
>> + * In reality, the remote may receive it intact, corrupted or not at all.
>> + * This could be called from atomic context as it simply
>> + * queues the data and returns a token (request_token_t)
>> + * against the request.
>> + * The client is later notified of successful transmission of
>> + * data over the channel via the 'txcb'. The client could in
>> + * turn queue more messages from txcb.
>> + */
>> +request_token_t ipc_send_message(void *channel, void *data)
>> +{
>> + struct ipc_chan *chan = (struct ipc_chan *)channel;
>> + request_token_t t;
>> +
>> + if (!chan)
>> + return 0;
>> +
>> + t = _add_to_rbuf(chan, data);
>> + if (!t)
>> + printk("Try increasing MBOX_TX_QUEUE_LEN\n");
>> +
>> + _msg_submit(chan);
>> +
>> + if (chan->txdone_method == TXDONE_BY_POLL)
>> + poll_txdone((unsigned long)chan->timer);
>> +
>> + if (chan->tx_block && chan->active_token) {
>
> Assigning the tx_block at channel request time might not be flexible
> enough. We should not impose that the client will request and release
> channels with different modes if both scenarios are needed.
>
If there is one blocking request, no more could arrive (blocking or
non-blocking). The increased code complexity doesn't justify the
rarely used feature. I don't see many use-cases when the client can't
release-request the channel. It's made like opening a file - blocking
vs non-blocking is the mode you open the resource.
>> + int ret;
>> + init_completion(&chan->tx_complete);
>> + ret = wait_for_completion_timeout(&chan->tx_complete,
>> + chan->tx_tout);
>> + if (ret == 0) {
>> + t = 0;
>> + tx_tick(chan, XFER_ERR);
>> + }
>> + }
>> +
>> + return t;
>> +}
>> +EXPORT_SYMBOL(ipc_send_message);
>
> Do we need to add an API similar to mailbox_msg_send_receive_no_irq, or
> are you leaving it to be client's responsibility?
>
Yes the API provides enough ways for a client to do that.
>> +
>> +/*
>> + * A client driver asks for exclusive use of a channel/mailbox.
>> + * If assigned, the channel has to be 'freed' before it could
>> + * be assigned to some other client.
>> + * After assignment, any packet received on this channel will be
>> + * handed over to the client via the 'rxcb' callback.
>> + * The 'txcb' callback is used to notify client upon sending the
>> + * packet over the channel, which may or may not have been yet
>> + * read by the remote processor.
>> + */
>> +void *ipc_request_channel(struct ipc_client *cl)
>> +{
>> + struct ipc_chan *chan;
>> + unsigned long flags;
>> + int ret = 0;
>> +
>> + mutex_lock(&chpool_mutex);
>> +
>> + list_for_each_entry(chan, &ipc_channels, node)
> I had a different design in mind here, maintain the list of controllers
> globally instead of the channels, that way your search can be a bit
> quicker.
>
Yeah, I changed that to it last minute :)
The API has no real use of controller nodes, it works solely on a
global pool of mailboxes. I tried to reflect that in code.
>> +
>> + if (txdone == TXDONE_BY_POLL) {
>> + timer = kzalloc(sizeof(struct tx_poll_timer), GFP_KERNEL);
>> + timer->period = ipc_con->txpoll_period;
>> + timer->poll.function = &poll_txdone;
>> + timer->poll.data = (unsigned long)timer;
>> + init_timer(&timer->poll);
>> + }
>> +
>> + channel = kzalloc(sizeof(struct ipc_chan) * num_links, GFP_KERNEL);
>
> minor, but you might be aware of this already, failure check needed
>
No, I overlooked. Thanks.
>> +
>> +static int __init ipc_mbox_init(void)
>> +{
>> + INIT_LIST_HEAD(&ipc_channels);
>> + return 0;
>> +}
>> +subsys_initcall(ipc_mbox_init);
>
> The ipc_mbox_init can be replaced by static list init of ipc_channels,
> nothing much to be gained by defining this function. We expect the
> mailbox.c to be built anyways whenever the mailbox framework is selected.
>
Yes, you are right. Thanks.
>> + * @cntlr_data: Optional controller specific parameters during channel request
>> + */
>> +struct ipc_client {
>> + char *chan_name;
>> + void (*rxcb)(void *data);
>> + void (*txcb)(request_token_t t, enum xfer_result r);
>> + bool tx_block;
>> + unsigned long tx_tout;
>> + bool knows_txdone;
>> + void *cntlr_data;
>
> What is the current use-case for exposing cntrl_data through ipc_client?
> I think this should be avoided and leave the controller configuration to
> the controller driver implementation. This will be a problem for
> multiple link scenarios.
>
There are some highly programmable controllers that, if the driver is
proper, could change behavior runtime. For ex, if the controller
supports, a client might want to broadcast messages to multiple
remotes or listen for messages from more than one remote source or
both. The client could, say, specify bitmasks for such source/sink
remotes via cntlr_data while requesting the mailbox. PL320 could work
in such mode if rest all fits.
>> +struct ipc_link {
>> + char link_name[16];
>> + void *api_priv;
>> +};
>
> this definitely needs a link-private void *ptr. PL320 doesn't have
> multiple links in the controller, but this is a problem for OMAP & DBX500.
>
The API keeps the ipc_link provided by the controller driver, which
could have it embedded in its private representation of the h/w links.
For ex, my controller has 6 links, and its structure contains 6
element array of 'struct myhw_link' which embeds an ipc_link. And I
use container_of().
>> +struct ipc_link_ops {
>> + int (*send_data)(struct ipc_link *link, void *data);
>> + int (*startup)(struct ipc_link *link, void *params);
>> + void (*shutdown)(struct ipc_link *link);
>> + bool (*last_tx_done)(struct ipc_link *link);
> minor comment, maybe rename this to something that indicates link
> busyness or readiness
>
The API gives the controller only one TX at a time, so I chose the
name. What do you suggest?
>> +};
>> +
>> +/**
>> + * struct ipc_controller - Controller of a class of communication links
>> + * @controller_name: Literal name of the controller.
>> + * @ops: Operators that work on each communication link
>> + * @links: Null terminated array of links.
>> + * @txdone_irq: Indicates if the controller can report to API when the
>> + * last transmitted data was read by the remote. Eg, if it has some
>> + * TX ACK irq.
>> + * @txdone_poll: If the controller can read but not report the TX done.
>> + * Eg, is some register shows the TX status but no interrupt rises.
>> + * Ignored if 'txdone_irq' is set.
>> + * @txpoll_period: If 'txdone_poll' is in effect, the API polls for
>> + * last TX's status after these many millisecs
>> + */
>> +struct ipc_controller {
>> + char controller_name[16];
> i think this can be avoided and use the underlying dev-name directly
> based on the controller device. Adding a struct device pointer would
> automatically allow you to use the name from the probe.
>
Ah ha... yes. Thanks.
>> + struct ipc_link_ops *ops;
> I think this should be a property of the individual link for
> flexibility. Right now, it probably doesn't make a difference (as seen
> from the current mailbox code), but the moment we have one link behaving
> differently this will make the ops implementation a bit messy.
>
> I think we also need a controller-specific ops, to put common stuff
> between multiple links within the controller.
>
A controller is assumed to be a bunch of identical links/mailboxes.
However the cntlr_data could still help the client specify mode in
which it wants a mailbox behave.
Thanks and Regards.
-Jassi
^ permalink raw reply [flat|nested] 90+ messages in thread* [RFC 2/3] mailbox: Introduce a new common API
2013-05-04 19:08 ` Jassi Brar
@ 2013-05-06 23:45 ` Suman Anna
-1 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-05-06 23:45 UTC (permalink / raw)
To: linux-arm-kernel
Hi Jassi,
On 05/04/2013 02:08 PM, Jassi Brar wrote:
> Hi Suman,
>
>> Anyway, here is a summary of the open points that we have:
>> 1. Atomic Callbacks:
>> The current code provides some sort of buffering on Tx, but imposes the
>> restriction that the clients do the buffering on Rx. This is main
>> concern for OMAP.
>
> I am afraid a common API can't do without buffering TX and it can't do
> by buffering RX.
Maybe both these are properties of the controller then.
>
> The client(s) can always generate TX requests at a rate greater than
> the API could transmit on the physical link. So as much as we dislike
> it, we have to buffer TX requests, otherwise N clients would.
The current code doesn't support N clients today anyway, and if they are
blocking mode on top of it, it would never need Tx buffering.
>
> OTOH Buffering received packets in the API doesn't help anybody, it
> only incurs unavoidable latencies on clients. Only clients, that need
> to take non-atomic actions upon RX, would need to buffer RX. Other
> clients should not suffer.
As I have explained for OMAP, it is about performing the bottom-half of
the Rx interrupt in the OMAP mailbox controller driver. The clients just
process the message (independent of inherent knowledge that they all
have to behave in a certain way to perform the bottom-half). A single
transport payload used for both Rx and Tx would make the state machine
very simple.
>
> IMHO if clients on OMAP need to buffer RX, let us keep it OMAP
> specific. If number of such platforms rise in future we could move
> that as an _optional_ helper API on top, that does RX buffering on
> behalf of clients ?
Yeah, that's my idea as well to put this in the OMAP mailbox controller
driver implementation. The IPC stacks on OMAP are quite established
(both in remoteproc and DSP/Bridge - different generations), so I cannot
afford to break the functionality of either of those. This already means
I have to export some API from my controller driver and there is no
other way or now.
>
>> 2. Support for Shared Clients AND Time-shared Clients:
>> As I said before, we need the framework to be flexible enough to support
>> shared clients. The shared clients may not be applicable for all h/w
>> controllers, where a client has to send a series of operations that
>> needs to be sent to the remote before anyone else can use it. This is a
>> functional integration aspect, and is dictated by the nature of the
>> remote. For the time-shared clients, the remote must have some implicit
>> message protocol where in the remote is able to identify the macro
>> usecase through a specific message. The framework should allow the case
>> of shared clients, with the clients doing their own message demuxing.
>>
> If the API provides shared ownership of a mailbox, it won't work for
> clients that need exclusive ownership (like 'server' side
> implementation of a protocol).
> OTOH if the API provides exclusive ownership, it is still possible to
> emulate shared ownership by simply publishing the mailbox handle (void
> *chan) globally. It works for all.
I am not saying provide this always. Have this dictated by the
controller or mailbox (look at my branch). The global publication works
well functionality-wise for Tx, but not so much for Rx. In anycase,
global publication will have its own set of problems - mostly you are
implying another layer of implementation that provides this sharing
capability (since they have to be outside of any single client).
>
>>
>> 3. Buffering/Size: I think we need to take a re-look at the whole tx
>> buffering mechanism. You are using an array that stores just the
>> pointers, which means that there are underlying assumptions that the
>> clients are keeping their buffer ptrs intact until the duration of the
>> transfer is done. This means that one cannot use local payload
>> variables for messages in the calling functions. I feel this is
>> unnecessary burden on the clients.
>>
> Most of the clients won't queue more than 1 request at a time. And
> then, isn't it only natural that clients don't mess with requests
> after submitting them ? I see mailbox clients working quite like I2C
> clients.
The OMAP use-cases do this today, so not a good assumption to make. The
OMAP IPC is built upon simple transmitting and processing received
messages separately, we will not be using blocking mode or tx callbacks.
I really don't need the complexity of having to introduce tx callbacks
just because I need to know if my pointer can be freed, as I said - it
eliminates the simplicity of using local variables unless the send API
is blocking. void * works for OMAP since my payloads are only 4 bytes by
themselves (this is a discussion that came up during the earlier series
as well), but this is not flexible enough on the clients-side generally.
>
>> Secondly, we may not need the logic around client knows_txdone and
>> tx_buffering together. I understand the need for buffering when you have
>> TX_POLL or TX_IRQ, but as far as TX_ACK is concerned, it is a client
>> protocol-level knowledge, and so the tx buffering logic can remain with
>> the client itself. This should simplify the code a little bit and get
>> rid of the ipc_client_txdone API.
>>
> Yeah I had it that way originally. But then I realized we could be
> running an 'ACK Packet' protocol over a controller that supports only
> POLL. In that case the controller's poll doesn't override client's
> knows_txdone because we want the client to cut-short the next poll
> delay as soon as it gets an ACK packet.
I think the same can be achieved from the Rx path upon an interrupt in
the specific controller, without having to introduce the knows_txdone.
It is knowledge intrinsic to a controller/mailbox. I really hope that
there are no controllers where you have to poll for Rx too :)
>
>> Looking at the current use-cases, I think OMAP might be the only one
>> which needs the buffering. The API that Loic added suggests that he
>> either sends a message and expects a response, or just sends a message
>> if the transport is free (I am not sure if he is using the API that uses
>> buffering currently). PL320 is the same as the first scenario that Loic has.
>>
> Yeah I too expect TX buffering to be very rare.
If so, can we eliminate this for the first pass, and leave it to the
controllers for now or dictate this based on a tx_buffering property? As
I see it, we do not yet have an agreement on the Tx buffering semantics.
>
>> The other point is regarding the size field, I am not convinced that we
>> can take this out, and leave it between the client and the controller
>> implementation. I think the mailbox core should at least perform a check
>> based on the max size published by a controller driver. Like in the
>> PL320 patch conversion, the for loop for writing the data - how does it
>> know that the provided pointer is atleast 7 ints, and not smaller than
>> that?
>>
> First of all, PL320 is a highly configurable and programmable IP.
> pl320.c is very specific to Highbank's usage, it is not usable as such
> on other platforms. There are many things hardcoded and tacitly
> assumed in the driver. I just changed the API, didn't add anything
> new.
OK, should we be renaming this file accordingly then until another
common user comes along? Also, an appropriate 'depends on' in the Kconfig.
>
> Secondly, I believe we acknowledged the fact that a client can't do
> without controller specific knowledge. So in proper implementation the
> controller's header would look like
>
> struct foobar_con_mssg {
> int datalen; /* size of received packet in bytes */
> u8 *buf; /* buffer that holds the packet data */
> .....
> };
>
> So instead of telling the API the 'datalen' and the remaining
> structure of each message, we simply pass "struct foobar_con_mssg *"
> as a void* to the API.
This has to be documented very well. And yes, I did acknowledge the
inter-dependencies portion for understanding the packet structure (there
is no way if you have a packet of 6 ints and you expect specific ints
for specific fields). The size and void * are generic enough that can be
present in the core to perform some error checking. The API would be
akin to memcpy then. The other motivation is based on the buffering
scheme (same comment as above on Tx buffering), which we haven't agreed
upon completely.
>
>
>>> +
>>> +/*
>>> + * Called by a client to "put data on the h/w channel" so that if
>>> + * everything else is fine we don't need to do anything more locally
>>> + * for the remote to receive the data intact.
>>> + * In reality, the remote may receive it intact, corrupted or not at all.
>>> + * This could be called from atomic context as it simply
>>> + * queues the data and returns a token (request_token_t)
>>> + * against the request.
>>> + * The client is later notified of successful transmission of
>>> + * data over the channel via the 'txcb'. The client could in
>>> + * turn queue more messages from txcb.
>>> + */
>>> +request_token_t ipc_send_message(void *channel, void *data)
>>> +{
>>> + struct ipc_chan *chan = (struct ipc_chan *)channel;
>>> + request_token_t t;
>>> +
>>> + if (!chan)
>>> + return 0;
>>> +
>>> + t = _add_to_rbuf(chan, data);
>>> + if (!t)
>>> + printk("Try increasing MBOX_TX_QUEUE_LEN\n");
>>> +
>>> + _msg_submit(chan);
>>> +
>>> + if (chan->txdone_method == TXDONE_BY_POLL)
>>> + poll_txdone((unsigned long)chan->timer);
>>> +
>>> + if (chan->tx_block && chan->active_token) {
>>
>> Assigning the tx_block at channel request time might not be flexible
>> enough. We should not impose that the client will request and release
>> channels with different modes if both scenarios are needed.
>>
> If there is one blocking request, no more could arrive (blocking or
> non-blocking).
Well, you are assuming that there is only one thread from within the
client driver who is gonna use the send API. There is nothing in the
code that stops from say two threads/contexts queueing in the message
from the same client driver. I am ok from my side for this to be sorted
out later, since I won't be using the blocking mode, but I do have two
threads (in DSPBridge driver) that will do a send.
The increased code complexity doesn't justify the
> rarely used feature. I don't see many use-cases when the client can't
> release-request the channel. It's made like opening a file - blocking
> vs non-blocking is the mode you open the resource.
>
>>> + int ret;
>>> + init_completion(&chan->tx_complete);
>>> + ret = wait_for_completion_timeout(&chan->tx_complete,
>>> + chan->tx_tout);
>>> + if (ret == 0) {
>>> + t = 0;
>>> + tx_tick(chan, XFER_ERR);
>>> + }
>>> + }
>>> +
>>> + return t;
>>> +}
>>> +EXPORT_SYMBOL(ipc_send_message);
>>
>> Do we need to add an API similar to mailbox_msg_send_receive_no_irq, or
>> are you leaving it to be client's responsibility?
>>
> Yes the API provides enough ways for a client to do that.
I will let Loic comment on this, based on his usage/complexity in the
client. I think you got lucky with PL320 implementation since it fills
in the buffer on a Tx done, so you have folded that functionality in
send_data itself.
>
>>> +void *ipc_request_channel(struct ipc_client *cl)
>>> +{
>>> + struct ipc_chan *chan;
>>> + unsigned long flags;
>>> + int ret = 0;
>>> +
>>> + mutex_lock(&chpool_mutex);
>>> +
>>> + list_for_each_entry(chan, &ipc_channels, node)
>> I had a different design in mind here, maintain the list of controllers
>> globally instead of the channels, that way your search can be a bit
>> quicker.
>>
> Yeah, I changed that to it last minute :)
> The API has no real use of controller nodes, it works solely on a
> global pool of mailboxes. I tried to reflect that in code.
Can we go back to using controller nodes? I understand that the API
doesn't use much, but the list of controller nodes is gonna be static
anyway in the file. It just improves the search algorithm when looking
for a mailbox. We have a new device coming up (u-boot patches already
submitted) wherein we have 13 controller instances, each with about 4 to
6 links, a linear search would be really painful.
>
>
>>> + * @cntlr_data: Optional controller specific parameters during channel request
>>> + */
>>> +struct ipc_client {
>>> + char *chan_name;
>>> + void (*rxcb)(void *data);
>>> + void (*txcb)(request_token_t t, enum xfer_result r);
>>> + bool tx_block;
>>> + unsigned long tx_tout;
>>> + bool knows_txdone;
>>> + void *cntlr_data;
>>
>> What is the current use-case for exposing cntrl_data through ipc_client?
>> I think this should be avoided and leave the controller configuration to
>> the controller driver implementation. This will be a problem for
>> multiple link scenarios.
>>
> There are some highly programmable controllers that, if the driver is
> proper, could change behavior runtime. For ex, if the controller
> supports, a client might want to broadcast messages to multiple
> remotes or listen for messages from more than one remote source or
> both. The client could, say, specify bitmasks for such source/sink
> remotes via cntlr_data while requesting the mailbox. PL320 could work
> in such mode if rest all fits.
Are you configuring the specific mailbox or configuring the controller
on a whole in this particular example?
>
>
>>> +struct ipc_link {
>>> + char link_name[16];
>>> + void *api_priv;
>>> +};
>>
>> this definitely needs a link-private void *ptr. PL320 doesn't have
>> multiple links in the controller, but this is a problem for OMAP & DBX500.
>>
> The API keeps the ipc_link provided by the controller driver, which
> could have it embedded in its private representation of the h/w links.
> For ex, my controller has 6 links, and its structure contains 6
> element array of 'struct myhw_link' which embeds an ipc_link. And I
> use container_of().
Ofcourse, that works. But I was trying to avoid it, since it streamlines
the controller implementation better - it can avoid managing another
variable to pass the links just for registration (especially if I have
to dynamically create them), and also manage the creation/static
declaration of the actual links. The mailbox/ipc core is anyway
constructing a ipc_chan above this link, and doesn't use it directly for
anything.
>
>>> +struct ipc_link_ops {
>>> + int (*send_data)(struct ipc_link *link, void *data);
>>> + int (*startup)(struct ipc_link *link, void *params);
>>> + void (*shutdown)(struct ipc_link *link);
>>> + bool (*last_tx_done)(struct ipc_link *link);
>> minor comment, maybe rename this to something that indicates link
>> busyness or readiness
>>
> The API gives the controller only one TX at a time, so I chose the
> name. What do you suggest?
how about 'is_ready', i think it is a bit more generic name and
automatically implies there are no pending tx.
>>> +};
>>> +
>>> +/**
>>> + * struct ipc_controller - Controller of a class of communication links
>>> + * @controller_name: Literal name of the controller.
>>> + * @ops: Operators that work on each communication link
>>> + * @links: Null terminated array of links.
>>> + * @txdone_irq: Indicates if the controller can report to API when the
>>> + * last transmitted data was read by the remote. Eg, if it has some
>>> + * TX ACK irq.
>>> + * @txdone_poll: If the controller can read but not report the TX done.
>>> + * Eg, is some register shows the TX status but no interrupt rises.
>>> + * Ignored if 'txdone_irq' is set.
>>> + * @txpoll_period: If 'txdone_poll' is in effect, the API polls for
>>> + * last TX's status after these many millisecs
>>> + */
>>> +struct ipc_controller {
>>> + char controller_name[16];
>> i think this can be avoided and use the underlying dev-name directly
>> based on the controller device. Adding a struct device pointer would
>> automatically allow you to use the name from the probe.
>>
> Ah ha... yes. Thanks.
Is your intention to leave the dev out to the controller driver specific
structure (atleast from the OMAP example code). I was asking for this to
be part of the ipc_controller structure, use a struct device *dev
instead of controller_name, and construct the controller name from the
dev_name.
>
>>> + struct ipc_link_ops *ops;
>> I think this should be a property of the individual link for
>> flexibility. Right now, it probably doesn't make a difference (as seen
>> from the current mailbox code), but the moment we have one link behaving
>> differently this will make the ops implementation a bit messy.
>>
>> I think we also need a controller-specific ops, to put common stuff
>> between multiple links within the controller.
>>
> A controller is assumed to be a bunch of identical links/mailboxes.
> However the cntlr_data could still help the client specify mode in
> which it wants a mailbox behave.
Yeah, assumptions hold true until an unfortunate SoC comes up with it
:). The TI AM3352 has one link that doesn't behave quite like the other
(I am hoping to avoid using it), so I am ok with it for now. Guess we
can always change it when a concrete need arises.
I prefer to see the controller_ops for startup/shutdown atleast. This
will allow the controller code state machine to be much better
maintained and make the code look good. As I said, with our new device,
where we have 13 controllers this will be very helpful. Also from what
Andy explained, you seem to have two controllers that are quite
different from each other.
regards
Suman
^ permalink raw reply [flat|nested] 90+ messages in thread* Re: [RFC 2/3] mailbox: Introduce a new common API
@ 2013-05-06 23:45 ` Suman Anna
0 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-05-06 23:45 UTC (permalink / raw)
To: Jassi Brar
Cc: Jassi Brar, Loic PALLARDY (loic.pallardy@st.com), Arnd Bergmann,
lkml, linux-arm-kernel@lists.infradead.org, Andy Green
Hi Jassi,
On 05/04/2013 02:08 PM, Jassi Brar wrote:
> Hi Suman,
>
>> Anyway, here is a summary of the open points that we have:
>> 1. Atomic Callbacks:
>> The current code provides some sort of buffering on Tx, but imposes the
>> restriction that the clients do the buffering on Rx. This is main
>> concern for OMAP.
>
> I am afraid a common API can't do without buffering TX and it can't do
> by buffering RX.
Maybe both these are properties of the controller then.
>
> The client(s) can always generate TX requests at a rate greater than
> the API could transmit on the physical link. So as much as we dislike
> it, we have to buffer TX requests, otherwise N clients would.
The current code doesn't support N clients today anyway, and if they are
blocking mode on top of it, it would never need Tx buffering.
>
> OTOH Buffering received packets in the API doesn't help anybody, it
> only incurs unavoidable latencies on clients. Only clients, that need
> to take non-atomic actions upon RX, would need to buffer RX. Other
> clients should not suffer.
As I have explained for OMAP, it is about performing the bottom-half of
the Rx interrupt in the OMAP mailbox controller driver. The clients just
process the message (independent of inherent knowledge that they all
have to behave in a certain way to perform the bottom-half). A single
transport payload used for both Rx and Tx would make the state machine
very simple.
>
> IMHO if clients on OMAP need to buffer RX, let us keep it OMAP
> specific. If number of such platforms rise in future we could move
> that as an _optional_ helper API on top, that does RX buffering on
> behalf of clients ?
Yeah, that's my idea as well to put this in the OMAP mailbox controller
driver implementation. The IPC stacks on OMAP are quite established
(both in remoteproc and DSP/Bridge - different generations), so I cannot
afford to break the functionality of either of those. This already means
I have to export some API from my controller driver and there is no
other way or now.
>
>> 2. Support for Shared Clients AND Time-shared Clients:
>> As I said before, we need the framework to be flexible enough to support
>> shared clients. The shared clients may not be applicable for all h/w
>> controllers, where a client has to send a series of operations that
>> needs to be sent to the remote before anyone else can use it. This is a
>> functional integration aspect, and is dictated by the nature of the
>> remote. For the time-shared clients, the remote must have some implicit
>> message protocol where in the remote is able to identify the macro
>> usecase through a specific message. The framework should allow the case
>> of shared clients, with the clients doing their own message demuxing.
>>
> If the API provides shared ownership of a mailbox, it won't work for
> clients that need exclusive ownership (like 'server' side
> implementation of a protocol).
> OTOH if the API provides exclusive ownership, it is still possible to
> emulate shared ownership by simply publishing the mailbox handle (void
> *chan) globally. It works for all.
I am not saying provide this always. Have this dictated by the
controller or mailbox (look at my branch). The global publication works
well functionality-wise for Tx, but not so much for Rx. In anycase,
global publication will have its own set of problems - mostly you are
implying another layer of implementation that provides this sharing
capability (since they have to be outside of any single client).
>
>>
>> 3. Buffering/Size: I think we need to take a re-look at the whole tx
>> buffering mechanism. You are using an array that stores just the
>> pointers, which means that there are underlying assumptions that the
>> clients are keeping their buffer ptrs intact until the duration of the
>> transfer is done. This means that one cannot use local payload
>> variables for messages in the calling functions. I feel this is
>> unnecessary burden on the clients.
>>
> Most of the clients won't queue more than 1 request at a time. And
> then, isn't it only natural that clients don't mess with requests
> after submitting them ? I see mailbox clients working quite like I2C
> clients.
The OMAP use-cases do this today, so not a good assumption to make. The
OMAP IPC is built upon simple transmitting and processing received
messages separately, we will not be using blocking mode or tx callbacks.
I really don't need the complexity of having to introduce tx callbacks
just because I need to know if my pointer can be freed, as I said - it
eliminates the simplicity of using local variables unless the send API
is blocking. void * works for OMAP since my payloads are only 4 bytes by
themselves (this is a discussion that came up during the earlier series
as well), but this is not flexible enough on the clients-side generally.
>
>> Secondly, we may not need the logic around client knows_txdone and
>> tx_buffering together. I understand the need for buffering when you have
>> TX_POLL or TX_IRQ, but as far as TX_ACK is concerned, it is a client
>> protocol-level knowledge, and so the tx buffering logic can remain with
>> the client itself. This should simplify the code a little bit and get
>> rid of the ipc_client_txdone API.
>>
> Yeah I had it that way originally. But then I realized we could be
> running an 'ACK Packet' protocol over a controller that supports only
> POLL. In that case the controller's poll doesn't override client's
> knows_txdone because we want the client to cut-short the next poll
> delay as soon as it gets an ACK packet.
I think the same can be achieved from the Rx path upon an interrupt in
the specific controller, without having to introduce the knows_txdone.
It is knowledge intrinsic to a controller/mailbox. I really hope that
there are no controllers where you have to poll for Rx too :)
>
>> Looking at the current use-cases, I think OMAP might be the only one
>> which needs the buffering. The API that Loic added suggests that he
>> either sends a message and expects a response, or just sends a message
>> if the transport is free (I am not sure if he is using the API that uses
>> buffering currently). PL320 is the same as the first scenario that Loic has.
>>
> Yeah I too expect TX buffering to be very rare.
If so, can we eliminate this for the first pass, and leave it to the
controllers for now or dictate this based on a tx_buffering property? As
I see it, we do not yet have an agreement on the Tx buffering semantics.
>
>> The other point is regarding the size field, I am not convinced that we
>> can take this out, and leave it between the client and the controller
>> implementation. I think the mailbox core should at least perform a check
>> based on the max size published by a controller driver. Like in the
>> PL320 patch conversion, the for loop for writing the data - how does it
>> know that the provided pointer is atleast 7 ints, and not smaller than
>> that?
>>
> First of all, PL320 is a highly configurable and programmable IP.
> pl320.c is very specific to Highbank's usage, it is not usable as such
> on other platforms. There are many things hardcoded and tacitly
> assumed in the driver. I just changed the API, didn't add anything
> new.
OK, should we be renaming this file accordingly then until another
common user comes along? Also, an appropriate 'depends on' in the Kconfig.
>
> Secondly, I believe we acknowledged the fact that a client can't do
> without controller specific knowledge. So in proper implementation the
> controller's header would look like
>
> struct foobar_con_mssg {
> int datalen; /* size of received packet in bytes */
> u8 *buf; /* buffer that holds the packet data */
> .....
> };
>
> So instead of telling the API the 'datalen' and the remaining
> structure of each message, we simply pass "struct foobar_con_mssg *"
> as a void* to the API.
This has to be documented very well. And yes, I did acknowledge the
inter-dependencies portion for understanding the packet structure (there
is no way if you have a packet of 6 ints and you expect specific ints
for specific fields). The size and void * are generic enough that can be
present in the core to perform some error checking. The API would be
akin to memcpy then. The other motivation is based on the buffering
scheme (same comment as above on Tx buffering), which we haven't agreed
upon completely.
>
>
>>> +
>>> +/*
>>> + * Called by a client to "put data on the h/w channel" so that if
>>> + * everything else is fine we don't need to do anything more locally
>>> + * for the remote to receive the data intact.
>>> + * In reality, the remote may receive it intact, corrupted or not at all.
>>> + * This could be called from atomic context as it simply
>>> + * queues the data and returns a token (request_token_t)
>>> + * against the request.
>>> + * The client is later notified of successful transmission of
>>> + * data over the channel via the 'txcb'. The client could in
>>> + * turn queue more messages from txcb.
>>> + */
>>> +request_token_t ipc_send_message(void *channel, void *data)
>>> +{
>>> + struct ipc_chan *chan = (struct ipc_chan *)channel;
>>> + request_token_t t;
>>> +
>>> + if (!chan)
>>> + return 0;
>>> +
>>> + t = _add_to_rbuf(chan, data);
>>> + if (!t)
>>> + printk("Try increasing MBOX_TX_QUEUE_LEN\n");
>>> +
>>> + _msg_submit(chan);
>>> +
>>> + if (chan->txdone_method == TXDONE_BY_POLL)
>>> + poll_txdone((unsigned long)chan->timer);
>>> +
>>> + if (chan->tx_block && chan->active_token) {
>>
>> Assigning the tx_block at channel request time might not be flexible
>> enough. We should not impose that the client will request and release
>> channels with different modes if both scenarios are needed.
>>
> If there is one blocking request, no more could arrive (blocking or
> non-blocking).
Well, you are assuming that there is only one thread from within the
client driver who is gonna use the send API. There is nothing in the
code that stops from say two threads/contexts queueing in the message
from the same client driver. I am ok from my side for this to be sorted
out later, since I won't be using the blocking mode, but I do have two
threads (in DSPBridge driver) that will do a send.
The increased code complexity doesn't justify the
> rarely used feature. I don't see many use-cases when the client can't
> release-request the channel. It's made like opening a file - blocking
> vs non-blocking is the mode you open the resource.
>
>>> + int ret;
>>> + init_completion(&chan->tx_complete);
>>> + ret = wait_for_completion_timeout(&chan->tx_complete,
>>> + chan->tx_tout);
>>> + if (ret == 0) {
>>> + t = 0;
>>> + tx_tick(chan, XFER_ERR);
>>> + }
>>> + }
>>> +
>>> + return t;
>>> +}
>>> +EXPORT_SYMBOL(ipc_send_message);
>>
>> Do we need to add an API similar to mailbox_msg_send_receive_no_irq, or
>> are you leaving it to be client's responsibility?
>>
> Yes the API provides enough ways for a client to do that.
I will let Loic comment on this, based on his usage/complexity in the
client. I think you got lucky with PL320 implementation since it fills
in the buffer on a Tx done, so you have folded that functionality in
send_data itself.
>
>>> +void *ipc_request_channel(struct ipc_client *cl)
>>> +{
>>> + struct ipc_chan *chan;
>>> + unsigned long flags;
>>> + int ret = 0;
>>> +
>>> + mutex_lock(&chpool_mutex);
>>> +
>>> + list_for_each_entry(chan, &ipc_channels, node)
>> I had a different design in mind here, maintain the list of controllers
>> globally instead of the channels, that way your search can be a bit
>> quicker.
>>
> Yeah, I changed that to it last minute :)
> The API has no real use of controller nodes, it works solely on a
> global pool of mailboxes. I tried to reflect that in code.
Can we go back to using controller nodes? I understand that the API
doesn't use much, but the list of controller nodes is gonna be static
anyway in the file. It just improves the search algorithm when looking
for a mailbox. We have a new device coming up (u-boot patches already
submitted) wherein we have 13 controller instances, each with about 4 to
6 links, a linear search would be really painful.
>
>
>>> + * @cntlr_data: Optional controller specific parameters during channel request
>>> + */
>>> +struct ipc_client {
>>> + char *chan_name;
>>> + void (*rxcb)(void *data);
>>> + void (*txcb)(request_token_t t, enum xfer_result r);
>>> + bool tx_block;
>>> + unsigned long tx_tout;
>>> + bool knows_txdone;
>>> + void *cntlr_data;
>>
>> What is the current use-case for exposing cntrl_data through ipc_client?
>> I think this should be avoided and leave the controller configuration to
>> the controller driver implementation. This will be a problem for
>> multiple link scenarios.
>>
> There are some highly programmable controllers that, if the driver is
> proper, could change behavior runtime. For ex, if the controller
> supports, a client might want to broadcast messages to multiple
> remotes or listen for messages from more than one remote source or
> both. The client could, say, specify bitmasks for such source/sink
> remotes via cntlr_data while requesting the mailbox. PL320 could work
> in such mode if rest all fits.
Are you configuring the specific mailbox or configuring the controller
on a whole in this particular example?
>
>
>>> +struct ipc_link {
>>> + char link_name[16];
>>> + void *api_priv;
>>> +};
>>
>> this definitely needs a link-private void *ptr. PL320 doesn't have
>> multiple links in the controller, but this is a problem for OMAP & DBX500.
>>
> The API keeps the ipc_link provided by the controller driver, which
> could have it embedded in its private representation of the h/w links.
> For ex, my controller has 6 links, and its structure contains 6
> element array of 'struct myhw_link' which embeds an ipc_link. And I
> use container_of().
Ofcourse, that works. But I was trying to avoid it, since it streamlines
the controller implementation better - it can avoid managing another
variable to pass the links just for registration (especially if I have
to dynamically create them), and also manage the creation/static
declaration of the actual links. The mailbox/ipc core is anyway
constructing a ipc_chan above this link, and doesn't use it directly for
anything.
>
>>> +struct ipc_link_ops {
>>> + int (*send_data)(struct ipc_link *link, void *data);
>>> + int (*startup)(struct ipc_link *link, void *params);
>>> + void (*shutdown)(struct ipc_link *link);
>>> + bool (*last_tx_done)(struct ipc_link *link);
>> minor comment, maybe rename this to something that indicates link
>> busyness or readiness
>>
> The API gives the controller only one TX at a time, so I chose the
> name. What do you suggest?
how about 'is_ready', i think it is a bit more generic name and
automatically implies there are no pending tx.
>>> +};
>>> +
>>> +/**
>>> + * struct ipc_controller - Controller of a class of communication links
>>> + * @controller_name: Literal name of the controller.
>>> + * @ops: Operators that work on each communication link
>>> + * @links: Null terminated array of links.
>>> + * @txdone_irq: Indicates if the controller can report to API when the
>>> + * last transmitted data was read by the remote. Eg, if it has some
>>> + * TX ACK irq.
>>> + * @txdone_poll: If the controller can read but not report the TX done.
>>> + * Eg, is some register shows the TX status but no interrupt rises.
>>> + * Ignored if 'txdone_irq' is set.
>>> + * @txpoll_period: If 'txdone_poll' is in effect, the API polls for
>>> + * last TX's status after these many millisecs
>>> + */
>>> +struct ipc_controller {
>>> + char controller_name[16];
>> i think this can be avoided and use the underlying dev-name directly
>> based on the controller device. Adding a struct device pointer would
>> automatically allow you to use the name from the probe.
>>
> Ah ha... yes. Thanks.
Is your intention to leave the dev out to the controller driver specific
structure (atleast from the OMAP example code). I was asking for this to
be part of the ipc_controller structure, use a struct device *dev
instead of controller_name, and construct the controller name from the
dev_name.
>
>>> + struct ipc_link_ops *ops;
>> I think this should be a property of the individual link for
>> flexibility. Right now, it probably doesn't make a difference (as seen
>> from the current mailbox code), but the moment we have one link behaving
>> differently this will make the ops implementation a bit messy.
>>
>> I think we also need a controller-specific ops, to put common stuff
>> between multiple links within the controller.
>>
> A controller is assumed to be a bunch of identical links/mailboxes.
> However the cntlr_data could still help the client specify mode in
> which it wants a mailbox behave.
Yeah, assumptions hold true until an unfortunate SoC comes up with it
:). The TI AM3352 has one link that doesn't behave quite like the other
(I am hoping to avoid using it), so I am ok with it for now. Guess we
can always change it when a concrete need arises.
I prefer to see the controller_ops for startup/shutdown atleast. This
will allow the controller code state machine to be much better
maintained and make the code look good. As I said, with our new device,
where we have 13 controllers this will be very helpful. Also from what
Andy explained, you seem to have two controllers that are quite
different from each other.
regards
Suman
^ permalink raw reply [flat|nested] 90+ messages in thread* [RFC 2/3] mailbox: Introduce a new common API
2013-05-06 23:45 ` Suman Anna
@ 2013-05-07 7:40 ` Jassi Brar
-1 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-05-07 7:40 UTC (permalink / raw)
To: linux-arm-kernel
Hi Suman,
On 7 May 2013 05:15, Suman Anna <s-anna@ti.com> wrote:
>>
>> The client(s) can always generate TX requests at a rate greater than
>> the API could transmit on the physical link. So as much as we dislike
>> it, we have to buffer TX requests, otherwise N clients would.
>
> The current code doesn't support N clients today anyway, and if they are
> blocking mode on top of it, it would never need Tx buffering.
>
No, I meant if the API doesn't buffer TX requests, then N client
drivers that simply need to send 2 or more messages would have to
buffer them locally.
Also by buffering TX requests the API reduces TX latency by
submitting the next request atomically as soon as it knows the last TX
is done.
>>
>> IMHO if clients on OMAP need to buffer RX, let us keep it OMAP
>> specific. If number of such platforms rise in future we could move
>> that as an _optional_ helper API on top, that does RX buffering on
>> behalf of clients ?
>
> Yeah, that's my idea as well to put this in the OMAP mailbox controller
> driver implementation. The IPC stacks on OMAP are quite established
> (both in remoteproc and DSP/Bridge - different generations), so I cannot
> afford to break the functionality of either of those. This already means
> I have to export some API from my controller driver and there is no
> other way or now.
>
Yes, I anticipated that.
Though imho OMAP's mailbox client/controller drivers could have be better.
The controller has 8 instances of identical mailboxes/links which
operate independent of each other. It is the s/w implementation that
has tacit understanding of who sends/receives on which link. So
ideally a client driver should simply, say for omap3430, ask for
separate "mbox1" to read messages and "mbox0" to send messages to DSP.
The unnecessary logical coupling of 0th and 1st links seems to be the
cause of 'hacks' like mailbox_dis/enable_irq(), whereas it should have
been release/request of the RX mbox.
Also logically coupling the independent physical links makes the
controller driver not very portable - what if some other platform has
usage of 3 send+receive and 2 send-only communications of the 8
physical links? The controller h/w is perfectly capable of providing
that, but its driver is not.
Similarly something sure seems fishy about the client driver having
to explicitly save and restore registers of controller. Anyways, OMAP'
implementation of mailbox is a separate subject.
>>>
>> If the API provides shared ownership of a mailbox, it won't work for
>> clients that need exclusive ownership (like 'server' side
>> implementation of a protocol).
>> OTOH if the API provides exclusive ownership, it is still possible to
>> emulate shared ownership by simply publishing the mailbox handle (void
>> *chan) globally. It works for all.
>
> I am not saying provide this always. Have this dictated by the
> controller or mailbox (look at my branch). The global publication works
> well functionality-wise for Tx, but not so much for Rx.
>
Right now for RX the clients register a blocking notifier call within
the API, you could move that out into the same code that publishes the
TX handlle globally. I am a bit hesitant because I want to make sure
we don't push anything into the common API that isn't already done
right in OMAP's implementation.
> In anycase,
> global publication will have its own set of problems - mostly you are
> implying another layer of implementation that provides this sharing
> capability (since they have to be outside of any single client).
>
Yes. Though yet another way of doing it is for the controller driver
to populate N logical per physical link that needs to be shared.
BTW, there doesn't exist any code/dt that creates the "omap-rproc"
platform_device so I may have misunderstood, but it seems the OMAP
already acquires a mailbox only once and save it in the data structure
representing the remote, which should simply work as such. No?
>>>
>> Most of the clients won't queue more than 1 request at a time. And
>> then, isn't it only natural that clients don't mess with requests
>> after submitting them ? I see mailbox clients working quite like I2C
>> clients.
>
> The OMAP use-cases do this today, so not a good assumption to make. The
> OMAP IPC is built upon simple transmitting and processing received
> messages separately, we will not be using blocking mode or tx callbacks.
> I really don't need the complexity of having to introduce tx callbacks
> just because I need to know if my pointer can be freed, as I said - it
> eliminates the simplicity of using local variables unless the send API
> is blocking. void * works for OMAP since my payloads are only 4 bytes by
> themselves (this is a discussion that came up during the earlier series
> as well), but this is not flexible enough on the clients-side generally.
>
The OMAP simply pass by value a u32 as a message. Why do you think it
won't work ? (Please have a look at my implementation of the omap2
controller driver)
>>
>>> Secondly, we may not need the logic around client knows_txdone and
>>> tx_buffering together. I understand the need for buffering when you have
>>> TX_POLL or TX_IRQ, but as far as TX_ACK is concerned, it is a client
>>> protocol-level knowledge, and so the tx buffering logic can remain with
>>> the client itself. This should simplify the code a little bit and get
>>> rid of the ipc_client_txdone API.
>>>
>> Yeah I had it that way originally. But then I realized we could be
>> running an 'ACK Packet' protocol over a controller that supports only
>> POLL. In that case the controller's poll doesn't override client's
>> knows_txdone because we want the client to cut-short the next poll
>> delay as soon as it gets an ACK packet.
>
> I think the same can be achieved from the Rx path upon an interrupt in
> the specific controller, without having to introduce the knows_txdone.
>
RX may not only be responses, it could very well be a remote command
just before the response to the last local command. OR the RX
responses may be out of order. So it can't be tied to the RX
interrupt.
> It is knowledge intrinsic to a controller/mailbox. I really hope that
> there are no controllers where you have to poll for Rx too :)
>
No, not intrinsic to controller, but to the high level protocol.
Yes, I assume the RX to be always interrupt driven :)
>>
>>> Looking at the current use-cases, I think OMAP might be the only one
>>> which needs the buffering. The API that Loic added suggests that he
>>> either sends a message and expects a response, or just sends a message
>>> if the transport is free (I am not sure if he is using the API that uses
>>> buffering currently). PL320 is the same as the first scenario that Loic has.
>>>
>> Yeah I too expect TX buffering to be very rare.
>
> If so, can we eliminate this for the first pass, and leave it to the
> controllers for now or dictate this based on a tx_buffering property? As
> I see it, we do not yet have an agreement on the Tx buffering semantics.
>
As I said, among other advantages, TX buffering is needed to reduce
latencies. My platform's protocol isn't yet clear as to whether use
shared-memory (secure and non-secure versions) for not very large data
transfers or interrupt payload. So if we get good enough speed we
could save cost on dedicated memories.
I wrote the omap2 controller driver code to convey my proposal better.
If you could please tell me how TX buffering is a problem for OMAP, it
will help me understand your concern better.
>>>
>> First of all, PL320 is a highly configurable and programmable IP.
>> pl320.c is very specific to Highbank's usage, it is not usable as such
>> on other platforms. There are many things hardcoded and tacitly
>> assumed in the driver. I just changed the API, didn't add anything
>> new.
>
> OK, should we be renaming this file accordingly then until another
> common user comes along? Also, an appropriate 'depends on' in the Kconfig.
>
Yes, will do. Thanks.
>>
>> Secondly, I believe we acknowledged the fact that a client can't do
>> without controller specific knowledge. So in proper implementation the
>> controller's header would look like
>>
>> struct foobar_con_mssg {
>> int datalen; /* size of received packet in bytes */
>> u8 *buf; /* buffer that holds the packet data */
>> .....
>> };
>>
>> So instead of telling the API the 'datalen' and the remaining
>> structure of each message, we simply pass "struct foobar_con_mssg *"
>> as a void* to the API.
>
> This has to be documented very well. And yes, I did acknowledge the
> inter-dependencies portion for understanding the packet structure (there
> is no way if you have a packet of 6 ints and you expect specific ints
> for specific fields). The size and void * are generic enough that can be
> present in the core to perform some error checking. The API would be
> akin to memcpy then. The other motivation is based on the buffering
> scheme (same comment as above on Tx buffering), which we haven't agreed
> upon completely.
>
How could the core perform any meaningful sanity check? What if we
send/receive proper 'datalen' of garbage data? What if the client
sends a "Give me info in not more than 32-bytes at a time" command to
the remote and the remote has valid info worth only 16 bytes?
IMHO there is not much for core to verify in a packet in transit. The
introduced checks are not worth it and actually limit some scenarios
like above example.
>>>
>>> Assigning the tx_block at channel request time might not be flexible
>>> enough. We should not impose that the client will request and release
>>> channels with different modes if both scenarios are needed.
>>>
>> If there is one blocking request, no more could arrive (blocking or
>> non-blocking).
>
> Well, you are assuming that there is only one thread from within the
> client driver who is gonna use the send API. There is nothing in the
> code that stops from say two threads/contexts queueing in the message
> from the same client driver. I am ok from my side for this to be sorted
> out later, since I won't be using the blocking mode,
>
Thanks
>>>
>>> Do we need to add an API similar to mailbox_msg_send_receive_no_irq, or
>>> are you leaving it to be client's responsibility?
>>>
>> Yes the API provides enough ways for a client to do that.
>
> I will let Loic comment on this, based on his usage/complexity in the
> client. I think you got lucky with PL320 implementation since it fills
> in the buffer on a Tx done, so you have folded that functionality in
> send_data itself.
>
No, the pl320 models the links properly. The links could be used for
RX or TX independently.
Any RX channel will return data only from a remote. The TX returns
data in tx-buffer which may or may not be changed by the remote after
reading. The Highbank's protocol make use of that feature. Though I am
interested to know if Loic has some unmet requirements still.
>>>
>> Yeah, I changed that to it last minute :)
>> The API has no real use of controller nodes, it works solely on a
>> global pool of mailboxes. I tried to reflect that in code.
>
> Can we go back to using controller nodes? I understand that the API
> doesn't use much, but the list of controller nodes is gonna be static
> anyway in the file. It just improves the search algorithm when looking
> for a mailbox. We have a new device coming up (u-boot patches already
> submitted) wherein we have 13 controller instances, each with about 4 to
> 6 links, a linear search would be really painful.
>
OK I'll keep a local list of controllers.
>>>
>> There are some highly programmable controllers that, if the driver is
>> proper, could change behavior runtime. For ex, if the controller
>> supports, a client might want to broadcast messages to multiple
>> remotes or listen for messages from more than one remote source or
>> both. The client could, say, specify bitmasks for such source/sink
>> remotes via cntlr_data while requesting the mailbox. PL320 could work
>> in such mode if rest all fits.
>
> Are you configuring the specific mailbox or configuring the controller
> on a whole in this particular example?
>
The cntlr_data is to configure the link/mailbox, not the controller.
>>>
>> The API gives the controller only one TX at a time, so I chose the
>> name. What do you suggest?
>
> how about 'is_ready', i think it is a bit more generic name and
> automatically implies there are no pending tx.
>
OK I can live with that ;)
>
> Is your intention to leave the dev out to the controller driver specific
> structure (atleast from the OMAP example code). I was asking for this to
> be part of the ipc_controller structure, use a struct device *dev
> instead of controller_name, and construct the controller name from the
> dev_name.
>
Yes, the ipc_controller should have it. I didn't implement that yet in
v2 though.
>>>
>> A controller is assumed to be a bunch of identical links/mailboxes.
>> However the cntlr_data could still help the client specify mode in
>> which it wants a mailbox behave.
>
> Yeah, assumptions hold true until an unfortunate SoC comes up with it
> :). The TI AM3352 has one link that doesn't behave quite like the other
> (I am hoping to avoid using it), so I am ok with it for now. Guess we
> can always change it when a concrete need arises.
>
It would work right now :) If your controller has 2 types of links,
the driver should register each bunch separately, each with its own
ipc_link_ops, with the core. The core can already take care of the
rest.
> I prefer to see the controller_ops for startup/shutdown atleast. This
> will allow the controller code state machine to be much better
> maintained and make the code look good. As I said, with our new device,
> where we have 13 controllers this will be very helpful. Also from what
> Andy explained, you seem to have two controllers that are quite
> different from each other.
>
Yes, I have 2 types of controllers and I already thought about
multiple controllers in a platform.
What do you expect to do in controller startup/shutdown? Since the
API works on individual links, not the controllers, when would it call
controller_start() and controller_shutdown()? The ipc_link's
startup/shutdown are already non-atomic, if the controller needs any
setup it could do it in the first link's startup and teardown in the
last link's shutdown. Not to forget the resources lke IRQ are usually
per Link which should be acquired/released upon link's use/idle.
While writing the OMAP2 controller driver, did I miss any controller
startup/shutdown ?
Regards,
-Jassi
^ permalink raw reply [flat|nested] 90+ messages in thread* Re: [RFC 2/3] mailbox: Introduce a new common API
@ 2013-05-07 7:40 ` Jassi Brar
0 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-05-07 7:40 UTC (permalink / raw)
To: Suman Anna
Cc: Jassi Brar, Loic PALLARDY (loic.pallardy@st.com), Arnd Bergmann,
lkml, linux-arm-kernel@lists.infradead.org, Andy Green
Hi Suman,
On 7 May 2013 05:15, Suman Anna <s-anna@ti.com> wrote:
>>
>> The client(s) can always generate TX requests at a rate greater than
>> the API could transmit on the physical link. So as much as we dislike
>> it, we have to buffer TX requests, otherwise N clients would.
>
> The current code doesn't support N clients today anyway, and if they are
> blocking mode on top of it, it would never need Tx buffering.
>
No, I meant if the API doesn't buffer TX requests, then N client
drivers that simply need to send 2 or more messages would have to
buffer them locally.
Also by buffering TX requests the API reduces TX latency by
submitting the next request atomically as soon as it knows the last TX
is done.
>>
>> IMHO if clients on OMAP need to buffer RX, let us keep it OMAP
>> specific. If number of such platforms rise in future we could move
>> that as an _optional_ helper API on top, that does RX buffering on
>> behalf of clients ?
>
> Yeah, that's my idea as well to put this in the OMAP mailbox controller
> driver implementation. The IPC stacks on OMAP are quite established
> (both in remoteproc and DSP/Bridge - different generations), so I cannot
> afford to break the functionality of either of those. This already means
> I have to export some API from my controller driver and there is no
> other way or now.
>
Yes, I anticipated that.
Though imho OMAP's mailbox client/controller drivers could have be better.
The controller has 8 instances of identical mailboxes/links which
operate independent of each other. It is the s/w implementation that
has tacit understanding of who sends/receives on which link. So
ideally a client driver should simply, say for omap3430, ask for
separate "mbox1" to read messages and "mbox0" to send messages to DSP.
The unnecessary logical coupling of 0th and 1st links seems to be the
cause of 'hacks' like mailbox_dis/enable_irq(), whereas it should have
been release/request of the RX mbox.
Also logically coupling the independent physical links makes the
controller driver not very portable - what if some other platform has
usage of 3 send+receive and 2 send-only communications of the 8
physical links? The controller h/w is perfectly capable of providing
that, but its driver is not.
Similarly something sure seems fishy about the client driver having
to explicitly save and restore registers of controller. Anyways, OMAP'
implementation of mailbox is a separate subject.
>>>
>> If the API provides shared ownership of a mailbox, it won't work for
>> clients that need exclusive ownership (like 'server' side
>> implementation of a protocol).
>> OTOH if the API provides exclusive ownership, it is still possible to
>> emulate shared ownership by simply publishing the mailbox handle (void
>> *chan) globally. It works for all.
>
> I am not saying provide this always. Have this dictated by the
> controller or mailbox (look at my branch). The global publication works
> well functionality-wise for Tx, but not so much for Rx.
>
Right now for RX the clients register a blocking notifier call within
the API, you could move that out into the same code that publishes the
TX handlle globally. I am a bit hesitant because I want to make sure
we don't push anything into the common API that isn't already done
right in OMAP's implementation.
> In anycase,
> global publication will have its own set of problems - mostly you are
> implying another layer of implementation that provides this sharing
> capability (since they have to be outside of any single client).
>
Yes. Though yet another way of doing it is for the controller driver
to populate N logical per physical link that needs to be shared.
BTW, there doesn't exist any code/dt that creates the "omap-rproc"
platform_device so I may have misunderstood, but it seems the OMAP
already acquires a mailbox only once and save it in the data structure
representing the remote, which should simply work as such. No?
>>>
>> Most of the clients won't queue more than 1 request at a time. And
>> then, isn't it only natural that clients don't mess with requests
>> after submitting them ? I see mailbox clients working quite like I2C
>> clients.
>
> The OMAP use-cases do this today, so not a good assumption to make. The
> OMAP IPC is built upon simple transmitting and processing received
> messages separately, we will not be using blocking mode or tx callbacks.
> I really don't need the complexity of having to introduce tx callbacks
> just because I need to know if my pointer can be freed, as I said - it
> eliminates the simplicity of using local variables unless the send API
> is blocking. void * works for OMAP since my payloads are only 4 bytes by
> themselves (this is a discussion that came up during the earlier series
> as well), but this is not flexible enough on the clients-side generally.
>
The OMAP simply pass by value a u32 as a message. Why do you think it
won't work ? (Please have a look at my implementation of the omap2
controller driver)
>>
>>> Secondly, we may not need the logic around client knows_txdone and
>>> tx_buffering together. I understand the need for buffering when you have
>>> TX_POLL or TX_IRQ, but as far as TX_ACK is concerned, it is a client
>>> protocol-level knowledge, and so the tx buffering logic can remain with
>>> the client itself. This should simplify the code a little bit and get
>>> rid of the ipc_client_txdone API.
>>>
>> Yeah I had it that way originally. But then I realized we could be
>> running an 'ACK Packet' protocol over a controller that supports only
>> POLL. In that case the controller's poll doesn't override client's
>> knows_txdone because we want the client to cut-short the next poll
>> delay as soon as it gets an ACK packet.
>
> I think the same can be achieved from the Rx path upon an interrupt in
> the specific controller, without having to introduce the knows_txdone.
>
RX may not only be responses, it could very well be a remote command
just before the response to the last local command. OR the RX
responses may be out of order. So it can't be tied to the RX
interrupt.
> It is knowledge intrinsic to a controller/mailbox. I really hope that
> there are no controllers where you have to poll for Rx too :)
>
No, not intrinsic to controller, but to the high level protocol.
Yes, I assume the RX to be always interrupt driven :)
>>
>>> Looking at the current use-cases, I think OMAP might be the only one
>>> which needs the buffering. The API that Loic added suggests that he
>>> either sends a message and expects a response, or just sends a message
>>> if the transport is free (I am not sure if he is using the API that uses
>>> buffering currently). PL320 is the same as the first scenario that Loic has.
>>>
>> Yeah I too expect TX buffering to be very rare.
>
> If so, can we eliminate this for the first pass, and leave it to the
> controllers for now or dictate this based on a tx_buffering property? As
> I see it, we do not yet have an agreement on the Tx buffering semantics.
>
As I said, among other advantages, TX buffering is needed to reduce
latencies. My platform's protocol isn't yet clear as to whether use
shared-memory (secure and non-secure versions) for not very large data
transfers or interrupt payload. So if we get good enough speed we
could save cost on dedicated memories.
I wrote the omap2 controller driver code to convey my proposal better.
If you could please tell me how TX buffering is a problem for OMAP, it
will help me understand your concern better.
>>>
>> First of all, PL320 is a highly configurable and programmable IP.
>> pl320.c is very specific to Highbank's usage, it is not usable as such
>> on other platforms. There are many things hardcoded and tacitly
>> assumed in the driver. I just changed the API, didn't add anything
>> new.
>
> OK, should we be renaming this file accordingly then until another
> common user comes along? Also, an appropriate 'depends on' in the Kconfig.
>
Yes, will do. Thanks.
>>
>> Secondly, I believe we acknowledged the fact that a client can't do
>> without controller specific knowledge. So in proper implementation the
>> controller's header would look like
>>
>> struct foobar_con_mssg {
>> int datalen; /* size of received packet in bytes */
>> u8 *buf; /* buffer that holds the packet data */
>> .....
>> };
>>
>> So instead of telling the API the 'datalen' and the remaining
>> structure of each message, we simply pass "struct foobar_con_mssg *"
>> as a void* to the API.
>
> This has to be documented very well. And yes, I did acknowledge the
> inter-dependencies portion for understanding the packet structure (there
> is no way if you have a packet of 6 ints and you expect specific ints
> for specific fields). The size and void * are generic enough that can be
> present in the core to perform some error checking. The API would be
> akin to memcpy then. The other motivation is based on the buffering
> scheme (same comment as above on Tx buffering), which we haven't agreed
> upon completely.
>
How could the core perform any meaningful sanity check? What if we
send/receive proper 'datalen' of garbage data? What if the client
sends a "Give me info in not more than 32-bytes at a time" command to
the remote and the remote has valid info worth only 16 bytes?
IMHO there is not much for core to verify in a packet in transit. The
introduced checks are not worth it and actually limit some scenarios
like above example.
>>>
>>> Assigning the tx_block at channel request time might not be flexible
>>> enough. We should not impose that the client will request and release
>>> channels with different modes if both scenarios are needed.
>>>
>> If there is one blocking request, no more could arrive (blocking or
>> non-blocking).
>
> Well, you are assuming that there is only one thread from within the
> client driver who is gonna use the send API. There is nothing in the
> code that stops from say two threads/contexts queueing in the message
> from the same client driver. I am ok from my side for this to be sorted
> out later, since I won't be using the blocking mode,
>
Thanks
>>>
>>> Do we need to add an API similar to mailbox_msg_send_receive_no_irq, or
>>> are you leaving it to be client's responsibility?
>>>
>> Yes the API provides enough ways for a client to do that.
>
> I will let Loic comment on this, based on his usage/complexity in the
> client. I think you got lucky with PL320 implementation since it fills
> in the buffer on a Tx done, so you have folded that functionality in
> send_data itself.
>
No, the pl320 models the links properly. The links could be used for
RX or TX independently.
Any RX channel will return data only from a remote. The TX returns
data in tx-buffer which may or may not be changed by the remote after
reading. The Highbank's protocol make use of that feature. Though I am
interested to know if Loic has some unmet requirements still.
>>>
>> Yeah, I changed that to it last minute :)
>> The API has no real use of controller nodes, it works solely on a
>> global pool of mailboxes. I tried to reflect that in code.
>
> Can we go back to using controller nodes? I understand that the API
> doesn't use much, but the list of controller nodes is gonna be static
> anyway in the file. It just improves the search algorithm when looking
> for a mailbox. We have a new device coming up (u-boot patches already
> submitted) wherein we have 13 controller instances, each with about 4 to
> 6 links, a linear search would be really painful.
>
OK I'll keep a local list of controllers.
>>>
>> There are some highly programmable controllers that, if the driver is
>> proper, could change behavior runtime. For ex, if the controller
>> supports, a client might want to broadcast messages to multiple
>> remotes or listen for messages from more than one remote source or
>> both. The client could, say, specify bitmasks for such source/sink
>> remotes via cntlr_data while requesting the mailbox. PL320 could work
>> in such mode if rest all fits.
>
> Are you configuring the specific mailbox or configuring the controller
> on a whole in this particular example?
>
The cntlr_data is to configure the link/mailbox, not the controller.
>>>
>> The API gives the controller only one TX at a time, so I chose the
>> name. What do you suggest?
>
> how about 'is_ready', i think it is a bit more generic name and
> automatically implies there are no pending tx.
>
OK I can live with that ;)
>
> Is your intention to leave the dev out to the controller driver specific
> structure (atleast from the OMAP example code). I was asking for this to
> be part of the ipc_controller structure, use a struct device *dev
> instead of controller_name, and construct the controller name from the
> dev_name.
>
Yes, the ipc_controller should have it. I didn't implement that yet in
v2 though.
>>>
>> A controller is assumed to be a bunch of identical links/mailboxes.
>> However the cntlr_data could still help the client specify mode in
>> which it wants a mailbox behave.
>
> Yeah, assumptions hold true until an unfortunate SoC comes up with it
> :). The TI AM3352 has one link that doesn't behave quite like the other
> (I am hoping to avoid using it), so I am ok with it for now. Guess we
> can always change it when a concrete need arises.
>
It would work right now :) If your controller has 2 types of links,
the driver should register each bunch separately, each with its own
ipc_link_ops, with the core. The core can already take care of the
rest.
> I prefer to see the controller_ops for startup/shutdown atleast. This
> will allow the controller code state machine to be much better
> maintained and make the code look good. As I said, with our new device,
> where we have 13 controllers this will be very helpful. Also from what
> Andy explained, you seem to have two controllers that are quite
> different from each other.
>
Yes, I have 2 types of controllers and I already thought about
multiple controllers in a platform.
What do you expect to do in controller startup/shutdown? Since the
API works on individual links, not the controllers, when would it call
controller_start() and controller_shutdown()? The ipc_link's
startup/shutdown are already non-atomic, if the controller needs any
setup it could do it in the first link's startup and teardown in the
last link's shutdown. Not to forget the resources lke IRQ are usually
per Link which should be acquired/released upon link's use/idle.
While writing the OMAP2 controller driver, did I miss any controller
startup/shutdown ?
Regards,
-Jassi
^ permalink raw reply [flat|nested] 90+ messages in thread* [RFC 2/3] mailbox: Introduce a new common API
2013-05-07 7:40 ` Jassi Brar
@ 2013-05-07 21:48 ` Suman Anna
-1 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-05-07 21:48 UTC (permalink / raw)
To: linux-arm-kernel
Hi Jassi,
> On 7 May 2013 05:15, Suman Anna <s-anna@ti.com> wrote:
>>>
>>> The client(s) can always generate TX requests at a rate greater than
>>> the API could transmit on the physical link. So as much as we dislike
>>> it, we have to buffer TX requests, otherwise N clients would.
>>
>> The current code doesn't support N clients today anyway, and if they are
>> blocking mode on top of it, it would never need Tx buffering.
>>
> No, I meant if the API doesn't buffer TX requests, then N client
> drivers that simply need to send 2 or more messages would have to
> buffer them locally.
> Also by buffering TX requests the API reduces TX latency by
> submitting the next request atomically as soon as it knows the last TX
> is done.
OK. I do not have an issue with Tx buffering, but the manner it is done.
As I said before, there is a big implementation detail on client users
keeping their pointers in tact until the actual Tx is done - it
increases the complexity of client code. I have proposed an idea below
on the size discussion.
>>>
>>> IMHO if clients on OMAP need to buffer RX, let us keep it OMAP
>>> specific. If number of such platforms rise in future we could move
>>> that as an _optional_ helper API on top, that does RX buffering on
>>> behalf of clients ?
>>
>> Yeah, that's my idea as well to put this in the OMAP mailbox controller
>> driver implementation. The IPC stacks on OMAP are quite established
>> (both in remoteproc and DSP/Bridge - different generations), so I cannot
>> afford to break the functionality of either of those. This already means
>> I have to export some API from my controller driver and there is no
>> other way or now.
>>
> Yes, I anticipated that.
> Though imho OMAP's mailbox client/controller drivers could have be better.
> The controller has 8 instances of identical mailboxes/links which
> operate independent of each other. It is the s/w implementation that
> has tacit understanding of who sends/receives on which link. So
> ideally a client driver should simply, say for omap3430, ask for
> separate "mbox1" to read messages and "mbox0" to send messages to DSP.
I think this is a good example for link_ops specified with the link,
rather than the controller (send_data would be NULL on the Rx instance
for example). In anycase, there is not much to be bought implementing
this way on OMAP (maybe sometime in the far future if we assign receive
queues by processor, rather than defining the pair as a communication
link with some message decoding indicating tx processor), since our
remotes are processors and the link assignment configuration doesn't
change dynamically.
> The unnecessary logical coupling of 0th and 1st links seems to be the
> cause of 'hacks' like mailbox_dis/enable_irq(), whereas it should have
> been release/request of the RX mbox.
The logical coupling is not the reason, but that's because of the IPC
protocol stack implementation in DSP/Bridge, that's a story for another
day. remoteproc does not suffer from this.
> Also logically coupling the independent physical links makes the
> controller driver not very portable - what if some other platform has
> usage of 3 send+receive and 2 send-only communications of the 8
> physical links? The controller h/w is perfectly capable of providing
> that, but its driver is not.
That's upto that controller driver, right?
> Similarly something sure seems fishy about the client driver having
> to explicitly save and restore registers of controller. Anyways, OMAP'
> implementation of mailbox is a separate subject.
Yeah, not too concerned here, the save/restore part can be fixed rather
easily. I will look into that once we get the series and the initial
migration done.
>>>>
>>> If the API provides shared ownership of a mailbox, it won't work for
>>> clients that need exclusive ownership (like 'server' side
>>> implementation of a protocol).
>>> OTOH if the API provides exclusive ownership, it is still possible to
>>> emulate shared ownership by simply publishing the mailbox handle (void
>>> *chan) globally. It works for all.
>>
>> I am not saying provide this always. Have this dictated by the
>> controller or mailbox (look at my branch). The global publication works
>> well functionality-wise for Tx, but not so much for Rx.
>>
> Right now for RX the clients register a blocking notifier call within
> the API, you could move that out into the same code that publishes the
> TX handlle globally. I am a bit hesitant because I want to make sure
> we don't push anything into the common API that isn't already done
> right in OMAP's implementation.
>
>> In anycase,
>> global publication will have its own set of problems - mostly you are
>> implying another layer of implementation that provides this sharing
>> capability (since they have to be outside of any single client).
>>
> Yes. Though yet another way of doing it is for the controller driver
> to populate N logical per physical link that needs to be shared.
Yeah, I was implying the same idea.. The notifier chains in the current
driver really meant the logical and physical is the same as we are
returning the link handle, but with your current design of returning
ipc_chan as an opaque handle, you already have the base infrastructure
in place. Once we add tx callbacks, even the original mailbox series
would have transformed into having a similar concept as what you have.
>
> BTW, there doesn't exist any code/dt that creates the "omap-rproc"
> platform_device so I may have misunderstood, but it seems the OMAP
> already acquires a mailbox only once and save it in the data structure
> representing the remote, which should simply work as such. No?
Yeah, the rproc device creation code is currently out of tree (you can
find it in Ohad's remoteproc tree on git.kernel.org), it is currently
disabled due to the lack of multi-platform support in mailbox
(dependencies on this series and other DT conversions). The runtime
support would have relinquished the mailboxes. remoteproc already looks
in certain messages for device management, and others it passes onto the
rpmsg bus, but that is rather tightly coupled.
>
>
>>>>
>>> Most of the clients won't queue more than 1 request at a time. And
>>> then, isn't it only natural that clients don't mess with requests
>>> after submitting them ? I see mailbox clients working quite like I2C
>>> clients.
>>
>> The OMAP use-cases do this today, so not a good assumption to make. The
>> OMAP IPC is built upon simple transmitting and processing received
>> messages separately, we will not be using blocking mode or tx callbacks.
>> I really don't need the complexity of having to introduce tx callbacks
>> just because I need to know if my pointer can be freed, as I said - it
>> eliminates the simplicity of using local variables unless the send API
>> is blocking. void * works for OMAP since my payloads are only 4 bytes by
>> themselves (this is a discussion that came up during the earlier series
>> as well), but this is not flexible enough on the clients-side generally.
>>
> The OMAP simply pass by value a u32 as a message. Why do you think it
> won't work ? (Please have a look at my implementation of the omap2
> controller driver)
I am talking about queueing more than 1 request, we simply send a
message that can be triggered by either remoteproc for PM purposes or by
different rpmsg clients sending a message. It is delivered into the
mailbox and there is no implicit ack needed on tx_done since that is
dealt by the client messaging protocol sitting above. We will not be
using blocking mode, so there can be multiple requests at the same time
(in the queue).
>
>>>
>>>> Secondly, we may not need the logic around client knows_txdone and
>>>> tx_buffering together. I understand the need for buffering when you have
>>>> TX_POLL or TX_IRQ, but as far as TX_ACK is concerned, it is a client
>>>> protocol-level knowledge, and so the tx buffering logic can remain with
>>>> the client itself. This should simplify the code a little bit and get
>>>> rid of the ipc_client_txdone API.
>>>>
>>> Yeah I had it that way originally. But then I realized we could be
>>> running an 'ACK Packet' protocol over a controller that supports only
>>> POLL. In that case the controller's poll doesn't override client's
>>> knows_txdone because we want the client to cut-short the next poll
>>> delay as soon as it gets an ACK packet.
>>
>> I think the same can be achieved from the Rx path upon an interrupt in
>> the specific controller, without having to introduce the knows_txdone.
>>
> RX may not only be responses, it could very well be a remote command
> just before the response to the last local command. OR the RX
> responses may be out of order. So it can't be tied to the RX
> interrupt.
Do you have the same shared payload in both directions? If so, the above
will work. Otherwise, I understand the scenario.
>> It is knowledge intrinsic to a controller/mailbox. I really hope that
>> there are no controllers where you have to poll for Rx too :)
>>
> No, not intrinsic to controller, but to the high level protocol.
> Yes, I assume the RX to be always interrupt driven :)
Until someone (out of their minds) comes up with it :)
>
>>>
>>>> Looking at the current use-cases, I think OMAP might be the only one
>>>> which needs the buffering. The API that Loic added suggests that he
>>>> either sends a message and expects a response, or just sends a message
>>>> if the transport is free (I am not sure if he is using the API that uses
>>>> buffering currently). PL320 is the same as the first scenario that Loic has.
>>>>
>>> Yeah I too expect TX buffering to be very rare.
>>
>> If so, can we eliminate this for the first pass, and leave it to the
>> controllers for now or dictate this based on a tx_buffering property? As
>> I see it, we do not yet have an agreement on the Tx buffering semantics.
>>
> As I said, among other advantages, TX buffering is needed to reduce
> latencies. My platform's protocol isn't yet clear as to whether use
> shared-memory (secure and non-secure versions) for not very large data
> transfers or interrupt payload. So if we get good enough speed we
> could save cost on dedicated memories.
>
> I wrote the omap2 controller driver code to convey my proposal better.
> If you could please tell me how TX buffering is a problem for OMAP, it
> will help me understand your concern better.
It is not a problem for OMAP, it is about only maintaining the void * in
the Tx queue. Since OMAP uses only a u32 message, I could have used the
void * as the u32 message but not a pointer (same approach we had taken
in the previous series). I am talking in general about the void *, and
the limitations it poses on client users. The other thing is that you
have defined 10 as the queue length in the core, this should be dictated
by the controller or mailbox. What works for one platform may not work
for some others (this is what you saw as different config options
between OMAP and DBX500)
>>>
>>> Secondly, I believe we acknowledged the fact that a client can't do
>>> without controller specific knowledge. So in proper implementation the
>>> controller's header would look like
>>>
>>> struct foobar_con_mssg {
>>> int datalen; /* size of received packet in bytes */
>>> u8 *buf; /* buffer that holds the packet data */
>>> .....
>>> };
>>>
>>> So instead of telling the API the 'datalen' and the remaining
>>> structure of each message, we simply pass "struct foobar_con_mssg *"
>>> as a void* to the API.
>>
>> This has to be documented very well. And yes, I did acknowledge the
>> inter-dependencies portion for understanding the packet structure (there
>> is no way if you have a packet of 6 ints and you expect specific ints
>> for specific fields). The size and void * are generic enough that can be
>> present in the core to perform some error checking. The API would be
>> akin to memcpy then. The other motivation is based on the buffering
>> scheme (same comment as above on Tx buffering), which we haven't agreed
>> upon completely.
>>
> How could the core perform any meaningful sanity check? What if we
> send/receive proper 'datalen' of garbage data? What if the client
> sends a "Give me info in not more than 32-bytes at a time" command to
> the remote and the remote has valid info worth only 16 bytes?
> IMHO there is not much for core to verify in a packet in transit. The
> introduced checks are not worth it and actually limit some scenarios
> like above example.
OK, I had mainly two ideas, one telling the size of the void * on Tx,
with the controller driver specifying max (and maybe min sizes expected)
so that some base error check can be done. This may be a basic check,
but atleast provides some level of sanity checking. On Rx-side, send the
size of the void * back to the client driver. The example you gave is
upto the client communication protocol, the remote may not even choose
to send the data if it doesn't have 32 bytes.
The second (major reason) idea was driven by the kfifo implementation
for buffering in core code for which I need the size field, so that the
clients need not be intelligent about the status of its tx submission
and maintaining its pointer (if you have different calling contexts). I
guess both of us are coming from different perspectives here, you may
find kfifos inefficient and I may find the complexity in client driver
unnecessary for maintaining pointers. How about defining add and remove
buffer ops in the controller driver so that it can choose its
implementation, while we keep the overall flow architecture in the core
code?
>
>
>>>>
>>>> Assigning the tx_block at channel request time might not be flexible
>>>> enough. We should not impose that the client will request and release
>>>> channels with different modes if both scenarios are needed.
>>>>
>>> If there is one blocking request, no more could arrive (blocking or
>>> non-blocking).
>>
>> Well, you are assuming that there is only one thread from within the
>> client driver who is gonna use the send API. There is nothing in the
>> code that stops from say two threads/contexts queueing in the message
>> from the same client driver. I am ok from my side for this to be sorted
>> out later, since I won't be using the blocking mode,
>>
> Thanks
>
>>>>
>>>> Do we need to add an API similar to mailbox_msg_send_receive_no_irq, or
>>>> are you leaving it to be client's responsibility?
>>>>
>>> Yes the API provides enough ways for a client to do that.
>>
>> I will let Loic comment on this, based on his usage/complexity in the
>> client. I think you got lucky with PL320 implementation since it fills
>> in the buffer on a Tx done, so you have folded that functionality in
>> send_data itself.
>>
> No, the pl320 models the links properly. The links could be used for
> RX or TX independently.
> Any RX channel will return data only from a remote. The TX returns
> data in tx-buffer which may or may not be changed by the remote after
> reading. The Highbank's protocol make use of that feature. Though I am
> interested to know if Loic has some unmet requirements still.
thanks for the clarification.
>
>
>>>>
>>> Yeah, I changed that to it last minute :)
>>> The API has no real use of controller nodes, it works solely on a
>>> global pool of mailboxes. I tried to reflect that in code.
>>
>> Can we go back to using controller nodes? I understand that the API
>> doesn't use much, but the list of controller nodes is gonna be static
>> anyway in the file. It just improves the search algorithm when looking
>> for a mailbox. We have a new device coming up (u-boot patches already
>> submitted) wherein we have 13 controller instances, each with about 4 to
>> 6 links, a linear search would be really painful.
>>
> OK I'll keep a local list of controllers.
Sure, thanks.
>
>
>>>>
>>> There are some highly programmable controllers that, if the driver is
>>> proper, could change behavior runtime. For ex, if the controller
>>> supports, a client might want to broadcast messages to multiple
>>> remotes or listen for messages from more than one remote source or
>>> both. The client could, say, specify bitmasks for such source/sink
>>> remotes via cntlr_data while requesting the mailbox. PL320 could work
>>> in such mode if rest all fits.
>>
>> Are you configuring the specific mailbox or configuring the controller
>> on a whole in this particular example?
>>
> The cntlr_data is to configure the link/mailbox, not the controller.
then, can we rename this to 'link_data' or something that doesn't imply
controller to avoid confusion? I guess you meant it as controller driver
data rather than controller instance data, but the name is a bit misleading.
>
>>>>
>>> A controller is assumed to be a bunch of identical links/mailboxes.
>>> However the cntlr_data could still help the client specify mode in
>>> which it wants a mailbox behave.
>>
>> Yeah, assumptions hold true until an unfortunate SoC comes up with it
>> :). The TI AM3352 has one link that doesn't behave quite like the other
>> (I am hoping to avoid using it), so I am ok with it for now. Guess we
>> can always change it when a concrete need arises.
>>
> It would work right now :) If your controller has 2 types of links,
> the driver should register each bunch separately, each with its own
> ipc_link_ops, with the core. The core can already take care of the
> rest.
Yeah, it can be done that way, but that's a not-so-elegant work-around
IMHO. It's calling ipc_links_register twice on the same controller. Idea
was that the API would distinguish different controllers, not the same
one. Also, see the above example if you were to treat a link as Rx or Tx
only (functional behavior differences even though the link is exactly
the same type).
>
>> I prefer to see the controller_ops for startup/shutdown atleast. This
>> will allow the controller code state machine to be much better
>> maintained and make the code look good. As I said, with our new device,
>> where we have 13 controllers this will be very helpful. Also from what
>> Andy explained, you seem to have two controllers that are quite
>> different from each other.
>>
> Yes, I have 2 types of controllers and I already thought about
> multiple controllers in a platform.
> What do you expect to do in controller startup/shutdown? Since the
> API works on individual links, not the controllers, when would it call
> controller_start() and controller_shutdown()? The ipc_link's
> startup/shutdown are already non-atomic, if the controller needs any
> setup it could do it in the first link's startup and teardown in the
> last link's shutdown. Not to forget the resources lke IRQ are usually
> per Link which should be acquired/released upon link's use/idle.
> While writing the OMAP2 controller driver, did I miss any controller
> startup/shutdown ?
It just delineates the code better, and has flexibility later on to
extend any controller ops or exposing new controller-specific API. The
controller start and stop will be called in the same function as
ipc_request_channel. The other thing is adding a check for the presence
of startup ops before actually invoking it, that way controller
implementations with no special startup needed can avoid it. Controller
ops in ipc_controller and link_ops in ipc_link would give better
flexibility for controller driver implementations.
Yeah, the pm_runtime_enable cannot be called twice when mailbox is
invoked from multiple clients on separate links, so there has to a
controller-level logic/ref-counting for that. The clocking for us is on
the controller.
regards
Suman
^ permalink raw reply [flat|nested] 90+ messages in thread* Re: [RFC 2/3] mailbox: Introduce a new common API
@ 2013-05-07 21:48 ` Suman Anna
0 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-05-07 21:48 UTC (permalink / raw)
To: Jassi Brar
Cc: Jassi Brar, Loic PALLARDY (loic.pallardy@st.com), Arnd Bergmann,
lkml, linux-arm-kernel@lists.infradead.org, Andy Green
Hi Jassi,
> On 7 May 2013 05:15, Suman Anna <s-anna@ti.com> wrote:
>>>
>>> The client(s) can always generate TX requests at a rate greater than
>>> the API could transmit on the physical link. So as much as we dislike
>>> it, we have to buffer TX requests, otherwise N clients would.
>>
>> The current code doesn't support N clients today anyway, and if they are
>> blocking mode on top of it, it would never need Tx buffering.
>>
> No, I meant if the API doesn't buffer TX requests, then N client
> drivers that simply need to send 2 or more messages would have to
> buffer them locally.
> Also by buffering TX requests the API reduces TX latency by
> submitting the next request atomically as soon as it knows the last TX
> is done.
OK. I do not have an issue with Tx buffering, but the manner it is done.
As I said before, there is a big implementation detail on client users
keeping their pointers in tact until the actual Tx is done - it
increases the complexity of client code. I have proposed an idea below
on the size discussion.
>>>
>>> IMHO if clients on OMAP need to buffer RX, let us keep it OMAP
>>> specific. If number of such platforms rise in future we could move
>>> that as an _optional_ helper API on top, that does RX buffering on
>>> behalf of clients ?
>>
>> Yeah, that's my idea as well to put this in the OMAP mailbox controller
>> driver implementation. The IPC stacks on OMAP are quite established
>> (both in remoteproc and DSP/Bridge - different generations), so I cannot
>> afford to break the functionality of either of those. This already means
>> I have to export some API from my controller driver and there is no
>> other way or now.
>>
> Yes, I anticipated that.
> Though imho OMAP's mailbox client/controller drivers could have be better.
> The controller has 8 instances of identical mailboxes/links which
> operate independent of each other. It is the s/w implementation that
> has tacit understanding of who sends/receives on which link. So
> ideally a client driver should simply, say for omap3430, ask for
> separate "mbox1" to read messages and "mbox0" to send messages to DSP.
I think this is a good example for link_ops specified with the link,
rather than the controller (send_data would be NULL on the Rx instance
for example). In anycase, there is not much to be bought implementing
this way on OMAP (maybe sometime in the far future if we assign receive
queues by processor, rather than defining the pair as a communication
link with some message decoding indicating tx processor), since our
remotes are processors and the link assignment configuration doesn't
change dynamically.
> The unnecessary logical coupling of 0th and 1st links seems to be the
> cause of 'hacks' like mailbox_dis/enable_irq(), whereas it should have
> been release/request of the RX mbox.
The logical coupling is not the reason, but that's because of the IPC
protocol stack implementation in DSP/Bridge, that's a story for another
day. remoteproc does not suffer from this.
> Also logically coupling the independent physical links makes the
> controller driver not very portable - what if some other platform has
> usage of 3 send+receive and 2 send-only communications of the 8
> physical links? The controller h/w is perfectly capable of providing
> that, but its driver is not.
That's upto that controller driver, right?
> Similarly something sure seems fishy about the client driver having
> to explicitly save and restore registers of controller. Anyways, OMAP'
> implementation of mailbox is a separate subject.
Yeah, not too concerned here, the save/restore part can be fixed rather
easily. I will look into that once we get the series and the initial
migration done.
>>>>
>>> If the API provides shared ownership of a mailbox, it won't work for
>>> clients that need exclusive ownership (like 'server' side
>>> implementation of a protocol).
>>> OTOH if the API provides exclusive ownership, it is still possible to
>>> emulate shared ownership by simply publishing the mailbox handle (void
>>> *chan) globally. It works for all.
>>
>> I am not saying provide this always. Have this dictated by the
>> controller or mailbox (look at my branch). The global publication works
>> well functionality-wise for Tx, but not so much for Rx.
>>
> Right now for RX the clients register a blocking notifier call within
> the API, you could move that out into the same code that publishes the
> TX handlle globally. I am a bit hesitant because I want to make sure
> we don't push anything into the common API that isn't already done
> right in OMAP's implementation.
>
>> In anycase,
>> global publication will have its own set of problems - mostly you are
>> implying another layer of implementation that provides this sharing
>> capability (since they have to be outside of any single client).
>>
> Yes. Though yet another way of doing it is for the controller driver
> to populate N logical per physical link that needs to be shared.
Yeah, I was implying the same idea.. The notifier chains in the current
driver really meant the logical and physical is the same as we are
returning the link handle, but with your current design of returning
ipc_chan as an opaque handle, you already have the base infrastructure
in place. Once we add tx callbacks, even the original mailbox series
would have transformed into having a similar concept as what you have.
>
> BTW, there doesn't exist any code/dt that creates the "omap-rproc"
> platform_device so I may have misunderstood, but it seems the OMAP
> already acquires a mailbox only once and save it in the data structure
> representing the remote, which should simply work as such. No?
Yeah, the rproc device creation code is currently out of tree (you can
find it in Ohad's remoteproc tree on git.kernel.org), it is currently
disabled due to the lack of multi-platform support in mailbox
(dependencies on this series and other DT conversions). The runtime
support would have relinquished the mailboxes. remoteproc already looks
in certain messages for device management, and others it passes onto the
rpmsg bus, but that is rather tightly coupled.
>
>
>>>>
>>> Most of the clients won't queue more than 1 request at a time. And
>>> then, isn't it only natural that clients don't mess with requests
>>> after submitting them ? I see mailbox clients working quite like I2C
>>> clients.
>>
>> The OMAP use-cases do this today, so not a good assumption to make. The
>> OMAP IPC is built upon simple transmitting and processing received
>> messages separately, we will not be using blocking mode or tx callbacks.
>> I really don't need the complexity of having to introduce tx callbacks
>> just because I need to know if my pointer can be freed, as I said - it
>> eliminates the simplicity of using local variables unless the send API
>> is blocking. void * works for OMAP since my payloads are only 4 bytes by
>> themselves (this is a discussion that came up during the earlier series
>> as well), but this is not flexible enough on the clients-side generally.
>>
> The OMAP simply pass by value a u32 as a message. Why do you think it
> won't work ? (Please have a look at my implementation of the omap2
> controller driver)
I am talking about queueing more than 1 request, we simply send a
message that can be triggered by either remoteproc for PM purposes or by
different rpmsg clients sending a message. It is delivered into the
mailbox and there is no implicit ack needed on tx_done since that is
dealt by the client messaging protocol sitting above. We will not be
using blocking mode, so there can be multiple requests at the same time
(in the queue).
>
>>>
>>>> Secondly, we may not need the logic around client knows_txdone and
>>>> tx_buffering together. I understand the need for buffering when you have
>>>> TX_POLL or TX_IRQ, but as far as TX_ACK is concerned, it is a client
>>>> protocol-level knowledge, and so the tx buffering logic can remain with
>>>> the client itself. This should simplify the code a little bit and get
>>>> rid of the ipc_client_txdone API.
>>>>
>>> Yeah I had it that way originally. But then I realized we could be
>>> running an 'ACK Packet' protocol over a controller that supports only
>>> POLL. In that case the controller's poll doesn't override client's
>>> knows_txdone because we want the client to cut-short the next poll
>>> delay as soon as it gets an ACK packet.
>>
>> I think the same can be achieved from the Rx path upon an interrupt in
>> the specific controller, without having to introduce the knows_txdone.
>>
> RX may not only be responses, it could very well be a remote command
> just before the response to the last local command. OR the RX
> responses may be out of order. So it can't be tied to the RX
> interrupt.
Do you have the same shared payload in both directions? If so, the above
will work. Otherwise, I understand the scenario.
>> It is knowledge intrinsic to a controller/mailbox. I really hope that
>> there are no controllers where you have to poll for Rx too :)
>>
> No, not intrinsic to controller, but to the high level protocol.
> Yes, I assume the RX to be always interrupt driven :)
Until someone (out of their minds) comes up with it :)
>
>>>
>>>> Looking at the current use-cases, I think OMAP might be the only one
>>>> which needs the buffering. The API that Loic added suggests that he
>>>> either sends a message and expects a response, or just sends a message
>>>> if the transport is free (I am not sure if he is using the API that uses
>>>> buffering currently). PL320 is the same as the first scenario that Loic has.
>>>>
>>> Yeah I too expect TX buffering to be very rare.
>>
>> If so, can we eliminate this for the first pass, and leave it to the
>> controllers for now or dictate this based on a tx_buffering property? As
>> I see it, we do not yet have an agreement on the Tx buffering semantics.
>>
> As I said, among other advantages, TX buffering is needed to reduce
> latencies. My platform's protocol isn't yet clear as to whether use
> shared-memory (secure and non-secure versions) for not very large data
> transfers or interrupt payload. So if we get good enough speed we
> could save cost on dedicated memories.
>
> I wrote the omap2 controller driver code to convey my proposal better.
> If you could please tell me how TX buffering is a problem for OMAP, it
> will help me understand your concern better.
It is not a problem for OMAP, it is about only maintaining the void * in
the Tx queue. Since OMAP uses only a u32 message, I could have used the
void * as the u32 message but not a pointer (same approach we had taken
in the previous series). I am talking in general about the void *, and
the limitations it poses on client users. The other thing is that you
have defined 10 as the queue length in the core, this should be dictated
by the controller or mailbox. What works for one platform may not work
for some others (this is what you saw as different config options
between OMAP and DBX500)
>>>
>>> Secondly, I believe we acknowledged the fact that a client can't do
>>> without controller specific knowledge. So in proper implementation the
>>> controller's header would look like
>>>
>>> struct foobar_con_mssg {
>>> int datalen; /* size of received packet in bytes */
>>> u8 *buf; /* buffer that holds the packet data */
>>> .....
>>> };
>>>
>>> So instead of telling the API the 'datalen' and the remaining
>>> structure of each message, we simply pass "struct foobar_con_mssg *"
>>> as a void* to the API.
>>
>> This has to be documented very well. And yes, I did acknowledge the
>> inter-dependencies portion for understanding the packet structure (there
>> is no way if you have a packet of 6 ints and you expect specific ints
>> for specific fields). The size and void * are generic enough that can be
>> present in the core to perform some error checking. The API would be
>> akin to memcpy then. The other motivation is based on the buffering
>> scheme (same comment as above on Tx buffering), which we haven't agreed
>> upon completely.
>>
> How could the core perform any meaningful sanity check? What if we
> send/receive proper 'datalen' of garbage data? What if the client
> sends a "Give me info in not more than 32-bytes at a time" command to
> the remote and the remote has valid info worth only 16 bytes?
> IMHO there is not much for core to verify in a packet in transit. The
> introduced checks are not worth it and actually limit some scenarios
> like above example.
OK, I had mainly two ideas, one telling the size of the void * on Tx,
with the controller driver specifying max (and maybe min sizes expected)
so that some base error check can be done. This may be a basic check,
but atleast provides some level of sanity checking. On Rx-side, send the
size of the void * back to the client driver. The example you gave is
upto the client communication protocol, the remote may not even choose
to send the data if it doesn't have 32 bytes.
The second (major reason) idea was driven by the kfifo implementation
for buffering in core code for which I need the size field, so that the
clients need not be intelligent about the status of its tx submission
and maintaining its pointer (if you have different calling contexts). I
guess both of us are coming from different perspectives here, you may
find kfifos inefficient and I may find the complexity in client driver
unnecessary for maintaining pointers. How about defining add and remove
buffer ops in the controller driver so that it can choose its
implementation, while we keep the overall flow architecture in the core
code?
>
>
>>>>
>>>> Assigning the tx_block at channel request time might not be flexible
>>>> enough. We should not impose that the client will request and release
>>>> channels with different modes if both scenarios are needed.
>>>>
>>> If there is one blocking request, no more could arrive (blocking or
>>> non-blocking).
>>
>> Well, you are assuming that there is only one thread from within the
>> client driver who is gonna use the send API. There is nothing in the
>> code that stops from say two threads/contexts queueing in the message
>> from the same client driver. I am ok from my side for this to be sorted
>> out later, since I won't be using the blocking mode,
>>
> Thanks
>
>>>>
>>>> Do we need to add an API similar to mailbox_msg_send_receive_no_irq, or
>>>> are you leaving it to be client's responsibility?
>>>>
>>> Yes the API provides enough ways for a client to do that.
>>
>> I will let Loic comment on this, based on his usage/complexity in the
>> client. I think you got lucky with PL320 implementation since it fills
>> in the buffer on a Tx done, so you have folded that functionality in
>> send_data itself.
>>
> No, the pl320 models the links properly. The links could be used for
> RX or TX independently.
> Any RX channel will return data only from a remote. The TX returns
> data in tx-buffer which may or may not be changed by the remote after
> reading. The Highbank's protocol make use of that feature. Though I am
> interested to know if Loic has some unmet requirements still.
thanks for the clarification.
>
>
>>>>
>>> Yeah, I changed that to it last minute :)
>>> The API has no real use of controller nodes, it works solely on a
>>> global pool of mailboxes. I tried to reflect that in code.
>>
>> Can we go back to using controller nodes? I understand that the API
>> doesn't use much, but the list of controller nodes is gonna be static
>> anyway in the file. It just improves the search algorithm when looking
>> for a mailbox. We have a new device coming up (u-boot patches already
>> submitted) wherein we have 13 controller instances, each with about 4 to
>> 6 links, a linear search would be really painful.
>>
> OK I'll keep a local list of controllers.
Sure, thanks.
>
>
>>>>
>>> There are some highly programmable controllers that, if the driver is
>>> proper, could change behavior runtime. For ex, if the controller
>>> supports, a client might want to broadcast messages to multiple
>>> remotes or listen for messages from more than one remote source or
>>> both. The client could, say, specify bitmasks for such source/sink
>>> remotes via cntlr_data while requesting the mailbox. PL320 could work
>>> in such mode if rest all fits.
>>
>> Are you configuring the specific mailbox or configuring the controller
>> on a whole in this particular example?
>>
> The cntlr_data is to configure the link/mailbox, not the controller.
then, can we rename this to 'link_data' or something that doesn't imply
controller to avoid confusion? I guess you meant it as controller driver
data rather than controller instance data, but the name is a bit misleading.
>
>>>>
>>> A controller is assumed to be a bunch of identical links/mailboxes.
>>> However the cntlr_data could still help the client specify mode in
>>> which it wants a mailbox behave.
>>
>> Yeah, assumptions hold true until an unfortunate SoC comes up with it
>> :). The TI AM3352 has one link that doesn't behave quite like the other
>> (I am hoping to avoid using it), so I am ok with it for now. Guess we
>> can always change it when a concrete need arises.
>>
> It would work right now :) If your controller has 2 types of links,
> the driver should register each bunch separately, each with its own
> ipc_link_ops, with the core. The core can already take care of the
> rest.
Yeah, it can be done that way, but that's a not-so-elegant work-around
IMHO. It's calling ipc_links_register twice on the same controller. Idea
was that the API would distinguish different controllers, not the same
one. Also, see the above example if you were to treat a link as Rx or Tx
only (functional behavior differences even though the link is exactly
the same type).
>
>> I prefer to see the controller_ops for startup/shutdown atleast. This
>> will allow the controller code state machine to be much better
>> maintained and make the code look good. As I said, with our new device,
>> where we have 13 controllers this will be very helpful. Also from what
>> Andy explained, you seem to have two controllers that are quite
>> different from each other.
>>
> Yes, I have 2 types of controllers and I already thought about
> multiple controllers in a platform.
> What do you expect to do in controller startup/shutdown? Since the
> API works on individual links, not the controllers, when would it call
> controller_start() and controller_shutdown()? The ipc_link's
> startup/shutdown are already non-atomic, if the controller needs any
> setup it could do it in the first link's startup and teardown in the
> last link's shutdown. Not to forget the resources lke IRQ are usually
> per Link which should be acquired/released upon link's use/idle.
> While writing the OMAP2 controller driver, did I miss any controller
> startup/shutdown ?
It just delineates the code better, and has flexibility later on to
extend any controller ops or exposing new controller-specific API. The
controller start and stop will be called in the same function as
ipc_request_channel. The other thing is adding a check for the presence
of startup ops before actually invoking it, that way controller
implementations with no special startup needed can avoid it. Controller
ops in ipc_controller and link_ops in ipc_link would give better
flexibility for controller driver implementations.
Yeah, the pm_runtime_enable cannot be called twice when mailbox is
invoked from multiple clients on separate links, so there has to a
controller-level logic/ref-counting for that. The clocking for us is on
the controller.
regards
Suman
^ permalink raw reply [flat|nested] 90+ messages in thread* [RFC 2/3] mailbox: Introduce a new common API
2013-05-07 21:48 ` Suman Anna
@ 2013-05-08 5:44 ` Jassi Brar
-1 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-05-08 5:44 UTC (permalink / raw)
To: linux-arm-kernel
On 8 May 2013 03:18, Suman Anna <s-anna@ti.com> wrote:
> Hi Jassi,
>
>> On 7 May 2013 05:15, Suman Anna <s-anna@ti.com> wrote:
>>>>
>>>> The client(s) can always generate TX requests at a rate greater than
>>>> the API could transmit on the physical link. So as much as we dislike
>>>> it, we have to buffer TX requests, otherwise N clients would.
>>>
>>> The current code doesn't support N clients today anyway, and if they are
>>> blocking mode on top of it, it would never need Tx buffering.
>>>
>> No, I meant if the API doesn't buffer TX requests, then N client
>> drivers that simply need to send 2 or more messages would have to
>> buffer them locally.
>> Also by buffering TX requests the API reduces TX latency by
>> submitting the next request atomically as soon as it knows the last TX
>> is done.
>
> OK. I do not have an issue with Tx buffering,
>
OK, thanks for confirming.
> but the manner it is done.
> As I said before, there is a big implementation detail on client users
> keeping their pointers in tact until the actual Tx is done - it
> increases the complexity of client code. I have proposed an idea below
> on the size discussion.
>
What makes this API different that you want it to create internal
copies of clients' requests?
I2C clients have been working fine. I believe most other apis work
similarly. No client died of the burden of having to preserve request
pointers until the request is served by the API.
BTW, by forcing the API to not directly use request structures from
clients we kill chances of zero-copy transfer use-cases.
Sorry, I don't think just to make it "easy" for some rare client (that
issues non-blocking pass-by-reference requests and don't even care to
know if the message was ever sent), we should add complexity to the
API _and_ kill some important use-cases.
>> Also logically coupling the independent physical links makes the
>> controller driver not very portable - what if some other platform has
>> usage of 3 send+receive and 2 send-only communications of the 8
>> physical links? The controller h/w is perfectly capable of providing
>> that, but its driver is not.
>
> That's upto that controller driver, right?
>
Yes, I meant the OMAP's controller driver need to be re-written
properly so as to not couple 2 links together as one RX and one TX.
>>> In anycase,
>>> global publication will have its own set of problems - mostly you are
>>> implying another layer of implementation that provides this sharing
>>> capability (since they have to be outside of any single client).
>>>
>> Yes. Though yet another way of doing it is for the controller driver
>> to populate N logical per physical link that needs to be shared.
>
> Yeah, I was implying the same idea.. The notifier chains in the current
> driver really meant the logical and physical is the same as we are
> returning the link handle, but with your current design of returning
> ipc_chan as an opaque handle, you already have the base infrastructure
> in place. Once we add tx callbacks, even the original mailbox series
> would have transformed into having a similar concept as what you have.
>
Though I would still suggest you manage a remote by a local 'server'
code - not every client needs to be bothered with request/response for
every other client.
Also the OMAP might be lucky, but the remote may not always simply
respond to commands from local clients. The remote might also send
some SOS or similar command on behalf of the remote as a whole. Such
requests should be handled by that local 'server' code.
>> The OMAP simply pass by value a u32 as a message. Why do you think it
>> won't work ? (Please have a look at my implementation of the omap2
>> controller driver)
>
> I am talking about queueing more than 1 request, we simply send a
> message that can be triggered by either remoteproc for PM purposes or by
> different rpmsg clients sending a message. It is delivered into the
> mailbox and there is no implicit ack needed on tx_done since that is
> dealt by the client messaging protocol sitting above. We will not be
> using blocking mode, so there can be multiple requests at the same time
> (in the queue).
>
Yes that's what I am talking about. Multiple requests would still be
a u32 each.
Since for OMAP a message is just a 32-bit number, a client would queue
requests as
{
u32 cmd;
cmd = COMMAND_A;
ipc_send_message(chan, (void *)cmd);
cmd = COMMAND_B;
ipc_send_message(chan, (void *)cmd);
cmd = COMMAND_C;
ipc_send_message(chan, (void *)cmd);
}
The controller driver retrieves each message from the API and transmit as
writel((u32) voidptr_cmd, txreg);
How could it get any simpler ?
>>> It is knowledge intrinsic to a controller/mailbox. I really hope that
>>> there are no controllers where you have to poll for Rx too :)
>>>
>> No, not intrinsic to controller, but to the high level protocol.
>> Yes, I assume the RX to be always interrupt driven :)
>
> Until someone (out of their minds) comes up with it :)
>
OR someone smart tries to run some IPC stack over, say, gpio bitbanging ;)
>>>
>> As I said, among other advantages, TX buffering is needed to reduce
>> latencies. My platform's protocol isn't yet clear as to whether use
>> shared-memory (secure and non-secure versions) for not very large data
>> transfers or interrupt payload. So if we get good enough speed we
>> could save cost on dedicated memories.
>>
>> I wrote the omap2 controller driver code to convey my proposal better.
>> If you could please tell me how TX buffering is a problem for OMAP, it
>> will help me understand your concern better.
>
> It is not a problem for OMAP
>
Thanks for confirming that it's not a problem for OMAP.
> The other thing is that you
> have defined 10 as the queue length in the core, this should be dictated
> by the controller or mailbox. What works for one platform may not work
> for some others (this is what you saw as different config options
> between OMAP and DBX500)
>
Please read the multi-line comment over that single line define.
The buffer is cheap, just a void*, so we could make it even 50.
However I am not hung on that.
It purely depends upon how the client uses the link, so it can't be
driven by the controller. We could make it a Kconfig option. What do
you suggest ?
>>>
>> How could the core perform any meaningful sanity check? What if we
>> send/receive proper 'datalen' of garbage data? What if the client
>> sends a "Give me info in not more than 32-bytes at a time" command to
>> the remote and the remote has valid info worth only 16 bytes?
>> IMHO there is not much for core to verify in a packet in transit. The
>> introduced checks are not worth it and actually limit some scenarios
>> like above example.
>
> OK, I had mainly two ideas, one telling the size of the void * on Tx,
> with the controller driver specifying max (and maybe min sizes expected)
>
No, please. The controller driver is supposed to be reusable over
various protocols. A protocol dictates the length of each logical TX
unit. The max-min limits set by the controller driver might work for
one but fail some other higher level protocol.
>
> The second (major reason) idea was driven by the kfifo implementation
> for buffering in core code for which I need the size field, so that the
> clients need not be intelligent about the status of its tx submission
> and maintaining its pointer (if you have different calling contexts). I
> guess both of us are coming from different perspectives here, you may
> find kfifos inefficient and I may find the complexity in client driver
> unnecessary for maintaining pointers. How about defining add and remove
> buffer ops in the controller driver so that it can choose its
> implementation, while we keep the overall flow architecture in the core
> code?
>
As I said the API shouldn't keep internal copies of a client's
requests - it will only kill zero-copy use-cases while adding
complexity to the core.
Since it already works fine for OMAP, I request we leave it as such
for now at least.
>>>> There are some highly programmable controllers that, if the driver is
>>>> proper, could change behavior runtime. For ex, if the controller
>>>> supports, a client might want to broadcast messages to multiple
>>>> remotes or listen for messages from more than one remote source or
>>>> both. The client could, say, specify bitmasks for such source/sink
>>>> remotes via cntlr_data while requesting the mailbox. PL320 could work
>>>> in such mode if rest all fits.
>>>
>>> Are you configuring the specific mailbox or configuring the controller
>>> on a whole in this particular example?
>>>
>> The cntlr_data is to configure the link/mailbox, not the controller.
>
> then, can we rename this to 'link_data' or something that doesn't imply
> controller to avoid confusion? I guess you meant it as controller driver
> data rather than controller instance data, but the name is a bit misleading.
>
OK, I'll call it 'link_data' :)
>>>
>>> Yeah, assumptions hold true until an unfortunate SoC comes up with it
>>> :). The TI AM3352 has one link that doesn't behave quite like the other
>>> (I am hoping to avoid using it), so I am ok with it for now. Guess we
>>> can always change it when a concrete need arises.
>>>
>> It would work right now :) If your controller has 2 types of links,
>> the driver should register each bunch separately, each with its own
>> ipc_link_ops, with the core. The core can already take care of the
>> rest.
>
> Yeah, it can be done that way, but that's a not-so-elegant work-around
>
OTOH I find it elegant and not a "work around".
Liberally estimating, hardly 10% of IPC controllers would have 2
classes of physical links. Having link_ops per controller reduces the
number of ipc_link_ops structures 11:20 as compared to per link.
> IMHO. It's calling ipc_links_register twice on the same controller. Idea
> was that the API would distinguish different controllers, not the same
> one. Also, see the above example if you were to treat a link as Rx or Tx
> only (functional behavior differences even though the link is exactly
> the same type).
>
Why is calling ipc_links_register() twice, for a controller with 2
classes of links, a problem ?
And, No, the API need not know if the link is "RX" or "TX", which is
purely a logical function.
Please do tell me which controller physically limits its links to do
only "RX" or only "TX"? It's just the platform's firmware that
dictates who sends and who receives on which link, thereby giving the
illusion of a link being a "RX" or "TX" capable only.
Even if some h/w limits do exist on links of some controller, still
the API shouldn't try to discern a "RX" from a "TX" link. When a
client knows which link to open, it will also know if it should read
or write to it - this assumption already holds true. If the client
writes on a link that it's supposed to listen to remote on, no amount
of policing by the API could help.
Even if the API tries to differentiate "RX" and "TX" links, the
knowledge must come from a client (it's driven by the protocol on the
platform), and not the controller because each of its links are
physically equally capable to do RX or TX on each end.
>> Yes, I have 2 types of controllers and I already thought about
>> multiple controllers in a platform.
>> What do you expect to do in controller startup/shutdown? Since the
>> API works on individual links, not the controllers, when would it call
>> controller_start() and controller_shutdown()? The ipc_link's
>> startup/shutdown are already non-atomic, if the controller needs any
>> setup it could do it in the first link's startup and teardown in the
>> last link's shutdown. Not to forget the resources lke IRQ are usually
>> per Link which should be acquired/released upon link's use/idle.
>> While writing the OMAP2 controller driver, did I miss any controller
>> startup/shutdown ?
>
> It just delineates the code better, and has flexibility later on to
> extend any controller ops or exposing new controller-specific API. The
> controller start and stop will be called in the same function as
> ipc_request_channel.
>
Already one call ipc_link_ops.startup() reaches the controller upon
ipc_request_channel.
Do you mean ?
ipc_con_ops.startup();
ipc_link_ops.startup();
Why have 2 contiguous calls to the controller, to perform similar
functions? Not to forget the API and clients don't work on IPC
controllers, but only on links. So it would be out-of-line for them to
have to worry about the controller.
>
> Yeah, the pm_runtime_enable cannot be called twice when mailbox is
> invoked from multiple clients on separate links, so there has to a
> controller-level logic/ref-counting for that. The clocking for us is on
> the controller.
>
No. You could call pm_runtime_enable/disable any number of times as
long as they are balanced. The core does refcounting. Anyways, I am
not in favour of controller ops for other reasons mentioned above.
May I suggest, atm, we care only about things that don't work for OMAP
or STE. And leave the design issues for later when/if some real and
compelling enough need arises otherwise. I am afraid we are reaching
the point where this thread could be called, what Linus.T termed as,
"mental masxxxx" :O
Regards,
-Jassi
^ permalink raw reply [flat|nested] 90+ messages in thread* Re: [RFC 2/3] mailbox: Introduce a new common API
@ 2013-05-08 5:44 ` Jassi Brar
0 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-05-08 5:44 UTC (permalink / raw)
To: Suman Anna
Cc: Jassi Brar, Loic PALLARDY (loic.pallardy@st.com), Arnd Bergmann,
lkml, linux-arm-kernel@lists.infradead.org, Andy Green
On 8 May 2013 03:18, Suman Anna <s-anna@ti.com> wrote:
> Hi Jassi,
>
>> On 7 May 2013 05:15, Suman Anna <s-anna@ti.com> wrote:
>>>>
>>>> The client(s) can always generate TX requests at a rate greater than
>>>> the API could transmit on the physical link. So as much as we dislike
>>>> it, we have to buffer TX requests, otherwise N clients would.
>>>
>>> The current code doesn't support N clients today anyway, and if they are
>>> blocking mode on top of it, it would never need Tx buffering.
>>>
>> No, I meant if the API doesn't buffer TX requests, then N client
>> drivers that simply need to send 2 or more messages would have to
>> buffer them locally.
>> Also by buffering TX requests the API reduces TX latency by
>> submitting the next request atomically as soon as it knows the last TX
>> is done.
>
> OK. I do not have an issue with Tx buffering,
>
OK, thanks for confirming.
> but the manner it is done.
> As I said before, there is a big implementation detail on client users
> keeping their pointers in tact until the actual Tx is done - it
> increases the complexity of client code. I have proposed an idea below
> on the size discussion.
>
What makes this API different that you want it to create internal
copies of clients' requests?
I2C clients have been working fine. I believe most other apis work
similarly. No client died of the burden of having to preserve request
pointers until the request is served by the API.
BTW, by forcing the API to not directly use request structures from
clients we kill chances of zero-copy transfer use-cases.
Sorry, I don't think just to make it "easy" for some rare client (that
issues non-blocking pass-by-reference requests and don't even care to
know if the message was ever sent), we should add complexity to the
API _and_ kill some important use-cases.
>> Also logically coupling the independent physical links makes the
>> controller driver not very portable - what if some other platform has
>> usage of 3 send+receive and 2 send-only communications of the 8
>> physical links? The controller h/w is perfectly capable of providing
>> that, but its driver is not.
>
> That's upto that controller driver, right?
>
Yes, I meant the OMAP's controller driver need to be re-written
properly so as to not couple 2 links together as one RX and one TX.
>>> In anycase,
>>> global publication will have its own set of problems - mostly you are
>>> implying another layer of implementation that provides this sharing
>>> capability (since they have to be outside of any single client).
>>>
>> Yes. Though yet another way of doing it is for the controller driver
>> to populate N logical per physical link that needs to be shared.
>
> Yeah, I was implying the same idea.. The notifier chains in the current
> driver really meant the logical and physical is the same as we are
> returning the link handle, but with your current design of returning
> ipc_chan as an opaque handle, you already have the base infrastructure
> in place. Once we add tx callbacks, even the original mailbox series
> would have transformed into having a similar concept as what you have.
>
Though I would still suggest you manage a remote by a local 'server'
code - not every client needs to be bothered with request/response for
every other client.
Also the OMAP might be lucky, but the remote may not always simply
respond to commands from local clients. The remote might also send
some SOS or similar command on behalf of the remote as a whole. Such
requests should be handled by that local 'server' code.
>> The OMAP simply pass by value a u32 as a message. Why do you think it
>> won't work ? (Please have a look at my implementation of the omap2
>> controller driver)
>
> I am talking about queueing more than 1 request, we simply send a
> message that can be triggered by either remoteproc for PM purposes or by
> different rpmsg clients sending a message. It is delivered into the
> mailbox and there is no implicit ack needed on tx_done since that is
> dealt by the client messaging protocol sitting above. We will not be
> using blocking mode, so there can be multiple requests at the same time
> (in the queue).
>
Yes that's what I am talking about. Multiple requests would still be
a u32 each.
Since for OMAP a message is just a 32-bit number, a client would queue
requests as
{
u32 cmd;
cmd = COMMAND_A;
ipc_send_message(chan, (void *)cmd);
cmd = COMMAND_B;
ipc_send_message(chan, (void *)cmd);
cmd = COMMAND_C;
ipc_send_message(chan, (void *)cmd);
}
The controller driver retrieves each message from the API and transmit as
writel((u32) voidptr_cmd, txreg);
How could it get any simpler ?
>>> It is knowledge intrinsic to a controller/mailbox. I really hope that
>>> there are no controllers where you have to poll for Rx too :)
>>>
>> No, not intrinsic to controller, but to the high level protocol.
>> Yes, I assume the RX to be always interrupt driven :)
>
> Until someone (out of their minds) comes up with it :)
>
OR someone smart tries to run some IPC stack over, say, gpio bitbanging ;)
>>>
>> As I said, among other advantages, TX buffering is needed to reduce
>> latencies. My platform's protocol isn't yet clear as to whether use
>> shared-memory (secure and non-secure versions) for not very large data
>> transfers or interrupt payload. So if we get good enough speed we
>> could save cost on dedicated memories.
>>
>> I wrote the omap2 controller driver code to convey my proposal better.
>> If you could please tell me how TX buffering is a problem for OMAP, it
>> will help me understand your concern better.
>
> It is not a problem for OMAP
>
Thanks for confirming that it's not a problem for OMAP.
> The other thing is that you
> have defined 10 as the queue length in the core, this should be dictated
> by the controller or mailbox. What works for one platform may not work
> for some others (this is what you saw as different config options
> between OMAP and DBX500)
>
Please read the multi-line comment over that single line define.
The buffer is cheap, just a void*, so we could make it even 50.
However I am not hung on that.
It purely depends upon how the client uses the link, so it can't be
driven by the controller. We could make it a Kconfig option. What do
you suggest ?
>>>
>> How could the core perform any meaningful sanity check? What if we
>> send/receive proper 'datalen' of garbage data? What if the client
>> sends a "Give me info in not more than 32-bytes at a time" command to
>> the remote and the remote has valid info worth only 16 bytes?
>> IMHO there is not much for core to verify in a packet in transit. The
>> introduced checks are not worth it and actually limit some scenarios
>> like above example.
>
> OK, I had mainly two ideas, one telling the size of the void * on Tx,
> with the controller driver specifying max (and maybe min sizes expected)
>
No, please. The controller driver is supposed to be reusable over
various protocols. A protocol dictates the length of each logical TX
unit. The max-min limits set by the controller driver might work for
one but fail some other higher level protocol.
>
> The second (major reason) idea was driven by the kfifo implementation
> for buffering in core code for which I need the size field, so that the
> clients need not be intelligent about the status of its tx submission
> and maintaining its pointer (if you have different calling contexts). I
> guess both of us are coming from different perspectives here, you may
> find kfifos inefficient and I may find the complexity in client driver
> unnecessary for maintaining pointers. How about defining add and remove
> buffer ops in the controller driver so that it can choose its
> implementation, while we keep the overall flow architecture in the core
> code?
>
As I said the API shouldn't keep internal copies of a client's
requests - it will only kill zero-copy use-cases while adding
complexity to the core.
Since it already works fine for OMAP, I request we leave it as such
for now at least.
>>>> There are some highly programmable controllers that, if the driver is
>>>> proper, could change behavior runtime. For ex, if the controller
>>>> supports, a client might want to broadcast messages to multiple
>>>> remotes or listen for messages from more than one remote source or
>>>> both. The client could, say, specify bitmasks for such source/sink
>>>> remotes via cntlr_data while requesting the mailbox. PL320 could work
>>>> in such mode if rest all fits.
>>>
>>> Are you configuring the specific mailbox or configuring the controller
>>> on a whole in this particular example?
>>>
>> The cntlr_data is to configure the link/mailbox, not the controller.
>
> then, can we rename this to 'link_data' or something that doesn't imply
> controller to avoid confusion? I guess you meant it as controller driver
> data rather than controller instance data, but the name is a bit misleading.
>
OK, I'll call it 'link_data' :)
>>>
>>> Yeah, assumptions hold true until an unfortunate SoC comes up with it
>>> :). The TI AM3352 has one link that doesn't behave quite like the other
>>> (I am hoping to avoid using it), so I am ok with it for now. Guess we
>>> can always change it when a concrete need arises.
>>>
>> It would work right now :) If your controller has 2 types of links,
>> the driver should register each bunch separately, each with its own
>> ipc_link_ops, with the core. The core can already take care of the
>> rest.
>
> Yeah, it can be done that way, but that's a not-so-elegant work-around
>
OTOH I find it elegant and not a "work around".
Liberally estimating, hardly 10% of IPC controllers would have 2
classes of physical links. Having link_ops per controller reduces the
number of ipc_link_ops structures 11:20 as compared to per link.
> IMHO. It's calling ipc_links_register twice on the same controller. Idea
> was that the API would distinguish different controllers, not the same
> one. Also, see the above example if you were to treat a link as Rx or Tx
> only (functional behavior differences even though the link is exactly
> the same type).
>
Why is calling ipc_links_register() twice, for a controller with 2
classes of links, a problem ?
And, No, the API need not know if the link is "RX" or "TX", which is
purely a logical function.
Please do tell me which controller physically limits its links to do
only "RX" or only "TX"? It's just the platform's firmware that
dictates who sends and who receives on which link, thereby giving the
illusion of a link being a "RX" or "TX" capable only.
Even if some h/w limits do exist on links of some controller, still
the API shouldn't try to discern a "RX" from a "TX" link. When a
client knows which link to open, it will also know if it should read
or write to it - this assumption already holds true. If the client
writes on a link that it's supposed to listen to remote on, no amount
of policing by the API could help.
Even if the API tries to differentiate "RX" and "TX" links, the
knowledge must come from a client (it's driven by the protocol on the
platform), and not the controller because each of its links are
physically equally capable to do RX or TX on each end.
>> Yes, I have 2 types of controllers and I already thought about
>> multiple controllers in a platform.
>> What do you expect to do in controller startup/shutdown? Since the
>> API works on individual links, not the controllers, when would it call
>> controller_start() and controller_shutdown()? The ipc_link's
>> startup/shutdown are already non-atomic, if the controller needs any
>> setup it could do it in the first link's startup and teardown in the
>> last link's shutdown. Not to forget the resources lke IRQ are usually
>> per Link which should be acquired/released upon link's use/idle.
>> While writing the OMAP2 controller driver, did I miss any controller
>> startup/shutdown ?
>
> It just delineates the code better, and has flexibility later on to
> extend any controller ops or exposing new controller-specific API. The
> controller start and stop will be called in the same function as
> ipc_request_channel.
>
Already one call ipc_link_ops.startup() reaches the controller upon
ipc_request_channel.
Do you mean ?
ipc_con_ops.startup();
ipc_link_ops.startup();
Why have 2 contiguous calls to the controller, to perform similar
functions? Not to forget the API and clients don't work on IPC
controllers, but only on links. So it would be out-of-line for them to
have to worry about the controller.
>
> Yeah, the pm_runtime_enable cannot be called twice when mailbox is
> invoked from multiple clients on separate links, so there has to a
> controller-level logic/ref-counting for that. The clocking for us is on
> the controller.
>
No. You could call pm_runtime_enable/disable any number of times as
long as they are balanced. The core does refcounting. Anyways, I am
not in favour of controller ops for other reasons mentioned above.
May I suggest, atm, we care only about things that don't work for OMAP
or STE. And leave the design issues for later when/if some real and
compelling enough need arises otherwise. I am afraid we are reaching
the point where this thread could be called, what Linus.T termed as,
"mental masxxxx" :O
Regards,
-Jassi
^ permalink raw reply [flat|nested] 90+ messages in thread* [RFC 2/3] mailbox: Introduce a new common API
2013-05-08 5:44 ` Jassi Brar
@ 2013-05-09 1:25 ` Suman Anna
-1 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-05-09 1:25 UTC (permalink / raw)
To: linux-arm-kernel
Hi Jassi,
>>>>>
>>>>> The client(s) can always generate TX requests at a rate greater than
>>>>> the API could transmit on the physical link. So as much as we dislike
>>>>> it, we have to buffer TX requests, otherwise N clients would.
>>>>
>>>> The current code doesn't support N clients today anyway, and if they are
>>>> blocking mode on top of it, it would never need Tx buffering.
>>>>
>>> No, I meant if the API doesn't buffer TX requests, then N client
>>> drivers that simply need to send 2 or more messages would have to
>>> buffer them locally.
>>> Also by buffering TX requests the API reduces TX latency by
>>> submitting the next request atomically as soon as it knows the last TX
>>> is done.
>>
>> OK. I do not have an issue with Tx buffering,
>>
> OK, thanks for confirming.
>
>> but the manner it is done.
>> As I said before, there is a big implementation detail on client users
>> keeping their pointers in tact until the actual Tx is done - it
>> increases the complexity of client code. I have proposed an idea below
>> on the size discussion.
>>
> What makes this API different that you want it to create internal
> copies of clients' requests?
> I2C clients have been working fine. I believe most other apis work
> similarly. No client died of the burden of having to preserve request
> pointers until the request is served by the API.
You are implying that every client does need to implement a tx_callback,
or needs to care for because of the design in the common core code. I
cannot use local variables in whatever function I am sending the
requests. The example you have given below will fail when the client
doesn't care about tx callbacks neither is blocking, if the entire
request is implemented as a function.
>
> BTW, by forcing the API to not directly use request structures from
> clients we kill chances of zero-copy transfer use-cases.
I doubt true zero-copy transfers can be achieved for most controllers,
you will almost always copy the client payload into the physical
transport payload. For true zero-copy, you would have to give out the
ioremapped transport payload ptr to a client.
> Sorry, I don't think just to make it "easy" for some rare client (that
> issues non-blocking pass-by-reference requests and don't even care to
> know if the message was ever sent), we should add complexity to the
> API _and_ kill some important use-cases.
The OMAP usecase (existing now) uses this approach, and we got lucky
because our payload is u32 and we will overwrite the ptr with the actual
message value instead of using it as a ptr. The existing API will fail
if I just have these two conditions met by a client - no blocking and no
tx callbacks.
>>> Also logically coupling the independent physical links makes the
>>> controller driver not very portable - what if some other platform has
>>> usage of 3 send+receive and 2 send-only communications of the 8
>>> physical links? The controller h/w is perfectly capable of providing
>>> that, but its driver is not.
>>
>> That's upto that controller driver, right?
>>
> Yes, I meant the OMAP's controller driver need to be re-written
> properly so as to not couple 2 links together as one RX and one TX.
Maybe for a different type of controller, but OMAP definitely doesn't
need this additional complexity for the foreseeable future, neither in
the client nor in the controller driver.
>>>> In anycase,
>>>> global publication will have its own set of problems - mostly you are
>>>> implying another layer of implementation that provides this sharing
>>>> capability (since they have to be outside of any single client).
>>>>
>>> Yes. Though yet another way of doing it is for the controller driver
>>> to populate N logical per physical link that needs to be shared.
>>
>> Yeah, I was implying the same idea.. The notifier chains in the current
>> driver really meant the logical and physical is the same as we are
>> returning the link handle, but with your current design of returning
>> ipc_chan as an opaque handle, you already have the base infrastructure
>> in place. Once we add tx callbacks, even the original mailbox series
>> would have transformed into having a similar concept as what you have.
>>
> Though I would still suggest you manage a remote by a local 'server'
> code - not every client needs to be bothered with request/response for
> every other client.
That's a client protocol or functional integration aspect of the
clients, remote and controller driver on the particular SoC, right?
> Also the OMAP might be lucky, but the remote may not always simply
> respond to commands from local clients. The remote might also send
> some SOS or similar command on behalf of the remote as a whole. Such
> requests should be handled by that local 'server' code.
>
>
>>> The OMAP simply pass by value a u32 as a message. Why do you think it
>>> won't work ? (Please have a look at my implementation of the omap2
>>> controller driver)
>>
>> I am talking about queueing more than 1 request, we simply send a
>> message that can be triggered by either remoteproc for PM purposes or by
>> different rpmsg clients sending a message. It is delivered into the
>> mailbox and there is no implicit ack needed on tx_done since that is
>> dealt by the client messaging protocol sitting above. We will not be
>> using blocking mode, so there can be multiple requests at the same time
>> (in the queue).
>>
> Yes that's what I am talking about. Multiple requests would still be
> a u32 each.
This discussion came about from your statement, "Most of the clients
won't queue more than 1 request at a time. And then, isn't it only
natural that clients don't mess with requests after submitting them? I
see mailbox clients working quite like I2C clients."
> Since for OMAP a message is just a 32-bit number, a client would queue
> requests as
> {
> u32 cmd;
> cmd = COMMAND_A;
> ipc_send_message(chan, (void *)cmd);
> cmd = COMMAND_B;
> ipc_send_message(chan, (void *)cmd);
> cmd = COMMAND_C;
> ipc_send_message(chan, (void *)cmd);
> }
This is the example I mentioned above (Make cmd a structure or an
integer array).
>
>
>>>>
>>> As I said, among other advantages, TX buffering is needed to reduce
>>> latencies. My platform's protocol isn't yet clear as to whether use
>>> shared-memory (secure and non-secure versions) for not very large data
>>> transfers or interrupt payload. So if we get good enough speed we
>>> could save cost on dedicated memories.
>>>
>>> I wrote the omap2 controller driver code to convey my proposal better.
>>> If you could please tell me how TX buffering is a problem for OMAP, it
>>> will help me understand your concern better.
>>
>> It is not a problem for OMAP
>>
> Thanks for confirming that it's not a problem for OMAP.
>
>
>> The other thing is that you
>> have defined 10 as the queue length in the core, this should be dictated
>> by the controller or mailbox. What works for one platform may not work
>> for some others (this is what you saw as different config options
>> between OMAP and DBX500)
>>
> Please read the multi-line comment over that single line define.
> The buffer is cheap, just a void*, so we could make it even 50.
> However I am not hung on that.
OK good, because no single number would be good for all the controllers
or all possible clients.
> It purely depends upon how the client uses the link,
i guess you are saying this because of the possible blocking behavior of
a client, or the exclusive access to a link.
> so it can't be driven by the controller. We could make it a Kconfig option.
> What do you suggest?
I am saying controller/link because they are the ones that knows how the
physical transport is, and it may vary from one to another. I would
think that the client dependencies would become a subset of this.
Kconfig option is fine, but we have to think about the best way of
representing it from a multi-platform kernel support perspective.
>>>>
>>> How could the core perform any meaningful sanity check? What if we
>>> send/receive proper 'datalen' of garbage data? What if the client
>>> sends a "Give me info in not more than 32-bytes at a time" command to
>>> the remote and the remote has valid info worth only 16 bytes?
>>> IMHO there is not much for core to verify in a packet in transit. The
>>> introduced checks are not worth it and actually limit some scenarios
>>> like above example.
>>
>> OK, I had mainly two ideas, one telling the size of the void * on Tx,
>> with the controller driver specifying max (and maybe min sizes expected)
>>
> No, please. The controller driver is supposed to be reusable over
> various protocols. A protocol dictates the length of each logical TX
> unit. The max-min limits set by the controller driver might work for
> one but fail some other higher level protocol.
>
>>
>> The second (major reason) idea was driven by the kfifo implementation
>> for buffering in core code for which I need the size field, so that the
>> clients need not be intelligent about the status of its tx submission
>> and maintaining its pointer (if you have different calling contexts). I
>> guess both of us are coming from different perspectives here, you may
>> find kfifos inefficient and I may find the complexity in client driver
>> unnecessary for maintaining pointers. How about defining add and remove
>> buffer ops in the controller driver so that it can choose its
>> implementation, while we keep the overall flow architecture in the core
>> code?
>>
> As I said the API shouldn't keep internal copies of a client's
> requests - it will only kill zero-copy use-cases while adding
> complexity to the core.
This is the reason I was suggesting the nature be dictated by a
controller, rather than 1 design in core code mandating a certain usage.
The client & controller together would know the functional integration
aspect.
> Since it already works fine for OMAP, I request we leave it as such
> for now at least.
OK, we should make a note of it in some documentation.
>>>>
>>>> Yeah, assumptions hold true until an unfortunate SoC comes up with it
>>>> :). The TI AM3352 has one link that doesn't behave quite like the other
>>>> (I am hoping to avoid using it), so I am ok with it for now. Guess we
>>>> can always change it when a concrete need arises.
>>>>
>>> It would work right now :) If your controller has 2 types of links,
>>> the driver should register each bunch separately, each with its own
>>> ipc_link_ops, with the core. The core can already take care of the
>>> rest.
>>
>> Yeah, it can be done that way, but that's a not-so-elegant work-around
>>
> OTOH I find it elegant and not a "work around".
> Liberally estimating, hardly 10% of IPC controllers would have 2
> classes of physical links. Having link_ops per controller reduces the
> number of ipc_link_ops structures 11:20 as compared to per link.
>
>
>> IMHO. It's calling ipc_links_register twice on the same controller. Idea
>> was that the API would distinguish different controllers, not the same
>> one. Also, see the above example if you were to treat a link as Rx or Tx
>> only (functional behavior differences even though the link is exactly
>> the same type).
>>
> Why is calling ipc_links_register() twice, for a controller with 2
> classes of links, a problem ?
It will be inelegant, once you start maintaining the list of
ipc_controllers in the core code. You would have to search all the
controllers in the list with the matching name, and their links. You
will need to define multiple controller specific controller structures
and copy most of the elements from the actual h/w device into the
containing ipc_controller definition. As you said, the clients deal with
the links themselves, so making the ops part of the ipc_link definition
makes it easier on the controller implementations. You are anyway
caching the ops in ipc_chan (per link) and that is what you are using in
the core code. Majority of the time, you would be using the same ops for
all the links, but this gives the flexibility to links. You can avoid
having multiple controller instance denoting the h/w block, just because
of the difference in links behavior.
>
> And, No, the API need not know if the link is "RX" or "TX", which is
> purely a logical function.
>
> Please do tell me which controller physically limits its links to do
> only "RX" or only "TX"? It's just the platform's firmware that
> dictates who sends and who receives on which link, thereby giving the
> illusion of a link being a "RX" or "TX" capable only.
> Even if some h/w limits do exist on links of some controller, still
> the API shouldn't try to discern a "RX" from a "TX" link. When a
> client knows which link to open, it will also know if it should read
> or write to it - this assumption already holds true. If the client
> writes on a link that it's supposed to listen to remote on, no amount
> of policing by the API could help.
>
> Even if the API tries to differentiate "RX" and "TX" links, the
> knowledge must come from a client (it's driven by the protocol on the
> platform), and not the controller because each of its links are
> physically equally capable to do RX or TX on each end.
The API never differentiates, but the controller implementation has to.
>From a controller driver implementation perspective, you still have to
make sure that no client is calling an op on which it is not supposed
to. When you have a mixture like this, a common code would include some
dead paths for certain links.
>
>
>>> Yes, I have 2 types of controllers and I already thought about
>>> multiple controllers in a platform.
>>> What do you expect to do in controller startup/shutdown? Since the
>>> API works on individual links, not the controllers, when would it call
>>> controller_start() and controller_shutdown()? The ipc_link's
>>> startup/shutdown are already non-atomic, if the controller needs any
>>> setup it could do it in the first link's startup and teardown in the
>>> last link's shutdown. Not to forget the resources lke IRQ are usually
>>> per Link which should be acquired/released upon link's use/idle.
>>> While writing the OMAP2 controller driver, did I miss any controller
>>> startup/shutdown ?
>>
>> It just delineates the code better, and has flexibility later on to
>> extend any controller ops or exposing new controller-specific API. The
>> controller start and stop will be called in the same function as
>> ipc_request_channel.
>>
> Already one call ipc_link_ops.startup() reaches the controller upon
> ipc_request_channel.
>
> Do you mean ?
> ipc_con_ops.startup();
> ipc_link_ops.startup();
Yes.
>
> Why have 2 contiguous calls to the controller, to perform similar
> functions? Not to forget the API and clients don't work on IPC
> controllers, but only on links. So it would be out-of-line for them to
> have to worry about the controller.
As I said, mainly to help dilineate the controller implementation code
better. I can live with it not being present, as my startup function
would anyway be refactored into two - one for the controller, and one
for the link itself. And I bet most would do the same if they have
multiple links in a controller.
>
>>
>> Yeah, the pm_runtime_enable cannot be called twice when mailbox is
>> invoked from multiple clients on separate links, so there has to a
>> controller-level logic/ref-counting for that. The clocking for us is on
>> the controller.
>>
> No. You could call pm_runtime_enable/disable any number of times as
> long as they are balanced. The core does refcounting.
Exactly, as long as they are balanced. I have two clients dealing with
two remotes (using two links) so pm_runtime_enable on the h/w block
needs to be called only when the first one comes in. Anyway, this is
just an FYI, as this file will obviously be different since I have to
not break my existing clients.
regards
Suman
^ permalink raw reply [flat|nested] 90+ messages in thread* Re: [RFC 2/3] mailbox: Introduce a new common API
@ 2013-05-09 1:25 ` Suman Anna
0 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-05-09 1:25 UTC (permalink / raw)
To: Jassi Brar
Cc: Jassi Brar, Loic PALLARDY (loic.pallardy@st.com), Arnd Bergmann,
lkml, linux-arm-kernel@lists.infradead.org, Andy Green
Hi Jassi,
>>>>>
>>>>> The client(s) can always generate TX requests at a rate greater than
>>>>> the API could transmit on the physical link. So as much as we dislike
>>>>> it, we have to buffer TX requests, otherwise N clients would.
>>>>
>>>> The current code doesn't support N clients today anyway, and if they are
>>>> blocking mode on top of it, it would never need Tx buffering.
>>>>
>>> No, I meant if the API doesn't buffer TX requests, then N client
>>> drivers that simply need to send 2 or more messages would have to
>>> buffer them locally.
>>> Also by buffering TX requests the API reduces TX latency by
>>> submitting the next request atomically as soon as it knows the last TX
>>> is done.
>>
>> OK. I do not have an issue with Tx buffering,
>>
> OK, thanks for confirming.
>
>> but the manner it is done.
>> As I said before, there is a big implementation detail on client users
>> keeping their pointers in tact until the actual Tx is done - it
>> increases the complexity of client code. I have proposed an idea below
>> on the size discussion.
>>
> What makes this API different that you want it to create internal
> copies of clients' requests?
> I2C clients have been working fine. I believe most other apis work
> similarly. No client died of the burden of having to preserve request
> pointers until the request is served by the API.
You are implying that every client does need to implement a tx_callback,
or needs to care for because of the design in the common core code. I
cannot use local variables in whatever function I am sending the
requests. The example you have given below will fail when the client
doesn't care about tx callbacks neither is blocking, if the entire
request is implemented as a function.
>
> BTW, by forcing the API to not directly use request structures from
> clients we kill chances of zero-copy transfer use-cases.
I doubt true zero-copy transfers can be achieved for most controllers,
you will almost always copy the client payload into the physical
transport payload. For true zero-copy, you would have to give out the
ioremapped transport payload ptr to a client.
> Sorry, I don't think just to make it "easy" for some rare client (that
> issues non-blocking pass-by-reference requests and don't even care to
> know if the message was ever sent), we should add complexity to the
> API _and_ kill some important use-cases.
The OMAP usecase (existing now) uses this approach, and we got lucky
because our payload is u32 and we will overwrite the ptr with the actual
message value instead of using it as a ptr. The existing API will fail
if I just have these two conditions met by a client - no blocking and no
tx callbacks.
>>> Also logically coupling the independent physical links makes the
>>> controller driver not very portable - what if some other platform has
>>> usage of 3 send+receive and 2 send-only communications of the 8
>>> physical links? The controller h/w is perfectly capable of providing
>>> that, but its driver is not.
>>
>> That's upto that controller driver, right?
>>
> Yes, I meant the OMAP's controller driver need to be re-written
> properly so as to not couple 2 links together as one RX and one TX.
Maybe for a different type of controller, but OMAP definitely doesn't
need this additional complexity for the foreseeable future, neither in
the client nor in the controller driver.
>>>> In anycase,
>>>> global publication will have its own set of problems - mostly you are
>>>> implying another layer of implementation that provides this sharing
>>>> capability (since they have to be outside of any single client).
>>>>
>>> Yes. Though yet another way of doing it is for the controller driver
>>> to populate N logical per physical link that needs to be shared.
>>
>> Yeah, I was implying the same idea.. The notifier chains in the current
>> driver really meant the logical and physical is the same as we are
>> returning the link handle, but with your current design of returning
>> ipc_chan as an opaque handle, you already have the base infrastructure
>> in place. Once we add tx callbacks, even the original mailbox series
>> would have transformed into having a similar concept as what you have.
>>
> Though I would still suggest you manage a remote by a local 'server'
> code - not every client needs to be bothered with request/response for
> every other client.
That's a client protocol or functional integration aspect of the
clients, remote and controller driver on the particular SoC, right?
> Also the OMAP might be lucky, but the remote may not always simply
> respond to commands from local clients. The remote might also send
> some SOS or similar command on behalf of the remote as a whole. Such
> requests should be handled by that local 'server' code.
>
>
>>> The OMAP simply pass by value a u32 as a message. Why do you think it
>>> won't work ? (Please have a look at my implementation of the omap2
>>> controller driver)
>>
>> I am talking about queueing more than 1 request, we simply send a
>> message that can be triggered by either remoteproc for PM purposes or by
>> different rpmsg clients sending a message. It is delivered into the
>> mailbox and there is no implicit ack needed on tx_done since that is
>> dealt by the client messaging protocol sitting above. We will not be
>> using blocking mode, so there can be multiple requests at the same time
>> (in the queue).
>>
> Yes that's what I am talking about. Multiple requests would still be
> a u32 each.
This discussion came about from your statement, "Most of the clients
won't queue more than 1 request at a time. And then, isn't it only
natural that clients don't mess with requests after submitting them? I
see mailbox clients working quite like I2C clients."
> Since for OMAP a message is just a 32-bit number, a client would queue
> requests as
> {
> u32 cmd;
> cmd = COMMAND_A;
> ipc_send_message(chan, (void *)cmd);
> cmd = COMMAND_B;
> ipc_send_message(chan, (void *)cmd);
> cmd = COMMAND_C;
> ipc_send_message(chan, (void *)cmd);
> }
This is the example I mentioned above (Make cmd a structure or an
integer array).
>
>
>>>>
>>> As I said, among other advantages, TX buffering is needed to reduce
>>> latencies. My platform's protocol isn't yet clear as to whether use
>>> shared-memory (secure and non-secure versions) for not very large data
>>> transfers or interrupt payload. So if we get good enough speed we
>>> could save cost on dedicated memories.
>>>
>>> I wrote the omap2 controller driver code to convey my proposal better.
>>> If you could please tell me how TX buffering is a problem for OMAP, it
>>> will help me understand your concern better.
>>
>> It is not a problem for OMAP
>>
> Thanks for confirming that it's not a problem for OMAP.
>
>
>> The other thing is that you
>> have defined 10 as the queue length in the core, this should be dictated
>> by the controller or mailbox. What works for one platform may not work
>> for some others (this is what you saw as different config options
>> between OMAP and DBX500)
>>
> Please read the multi-line comment over that single line define.
> The buffer is cheap, just a void*, so we could make it even 50.
> However I am not hung on that.
OK good, because no single number would be good for all the controllers
or all possible clients.
> It purely depends upon how the client uses the link,
i guess you are saying this because of the possible blocking behavior of
a client, or the exclusive access to a link.
> so it can't be driven by the controller. We could make it a Kconfig option.
> What do you suggest?
I am saying controller/link because they are the ones that knows how the
physical transport is, and it may vary from one to another. I would
think that the client dependencies would become a subset of this.
Kconfig option is fine, but we have to think about the best way of
representing it from a multi-platform kernel support perspective.
>>>>
>>> How could the core perform any meaningful sanity check? What if we
>>> send/receive proper 'datalen' of garbage data? What if the client
>>> sends a "Give me info in not more than 32-bytes at a time" command to
>>> the remote and the remote has valid info worth only 16 bytes?
>>> IMHO there is not much for core to verify in a packet in transit. The
>>> introduced checks are not worth it and actually limit some scenarios
>>> like above example.
>>
>> OK, I had mainly two ideas, one telling the size of the void * on Tx,
>> with the controller driver specifying max (and maybe min sizes expected)
>>
> No, please. The controller driver is supposed to be reusable over
> various protocols. A protocol dictates the length of each logical TX
> unit. The max-min limits set by the controller driver might work for
> one but fail some other higher level protocol.
>
>>
>> The second (major reason) idea was driven by the kfifo implementation
>> for buffering in core code for which I need the size field, so that the
>> clients need not be intelligent about the status of its tx submission
>> and maintaining its pointer (if you have different calling contexts). I
>> guess both of us are coming from different perspectives here, you may
>> find kfifos inefficient and I may find the complexity in client driver
>> unnecessary for maintaining pointers. How about defining add and remove
>> buffer ops in the controller driver so that it can choose its
>> implementation, while we keep the overall flow architecture in the core
>> code?
>>
> As I said the API shouldn't keep internal copies of a client's
> requests - it will only kill zero-copy use-cases while adding
> complexity to the core.
This is the reason I was suggesting the nature be dictated by a
controller, rather than 1 design in core code mandating a certain usage.
The client & controller together would know the functional integration
aspect.
> Since it already works fine for OMAP, I request we leave it as such
> for now at least.
OK, we should make a note of it in some documentation.
>>>>
>>>> Yeah, assumptions hold true until an unfortunate SoC comes up with it
>>>> :). The TI AM3352 has one link that doesn't behave quite like the other
>>>> (I am hoping to avoid using it), so I am ok with it for now. Guess we
>>>> can always change it when a concrete need arises.
>>>>
>>> It would work right now :) If your controller has 2 types of links,
>>> the driver should register each bunch separately, each with its own
>>> ipc_link_ops, with the core. The core can already take care of the
>>> rest.
>>
>> Yeah, it can be done that way, but that's a not-so-elegant work-around
>>
> OTOH I find it elegant and not a "work around".
> Liberally estimating, hardly 10% of IPC controllers would have 2
> classes of physical links. Having link_ops per controller reduces the
> number of ipc_link_ops structures 11:20 as compared to per link.
>
>
>> IMHO. It's calling ipc_links_register twice on the same controller. Idea
>> was that the API would distinguish different controllers, not the same
>> one. Also, see the above example if you were to treat a link as Rx or Tx
>> only (functional behavior differences even though the link is exactly
>> the same type).
>>
> Why is calling ipc_links_register() twice, for a controller with 2
> classes of links, a problem ?
It will be inelegant, once you start maintaining the list of
ipc_controllers in the core code. You would have to search all the
controllers in the list with the matching name, and their links. You
will need to define multiple controller specific controller structures
and copy most of the elements from the actual h/w device into the
containing ipc_controller definition. As you said, the clients deal with
the links themselves, so making the ops part of the ipc_link definition
makes it easier on the controller implementations. You are anyway
caching the ops in ipc_chan (per link) and that is what you are using in
the core code. Majority of the time, you would be using the same ops for
all the links, but this gives the flexibility to links. You can avoid
having multiple controller instance denoting the h/w block, just because
of the difference in links behavior.
>
> And, No, the API need not know if the link is "RX" or "TX", which is
> purely a logical function.
>
> Please do tell me which controller physically limits its links to do
> only "RX" or only "TX"? It's just the platform's firmware that
> dictates who sends and who receives on which link, thereby giving the
> illusion of a link being a "RX" or "TX" capable only.
> Even if some h/w limits do exist on links of some controller, still
> the API shouldn't try to discern a "RX" from a "TX" link. When a
> client knows which link to open, it will also know if it should read
> or write to it - this assumption already holds true. If the client
> writes on a link that it's supposed to listen to remote on, no amount
> of policing by the API could help.
>
> Even if the API tries to differentiate "RX" and "TX" links, the
> knowledge must come from a client (it's driven by the protocol on the
> platform), and not the controller because each of its links are
> physically equally capable to do RX or TX on each end.
The API never differentiates, but the controller implementation has to.
>From a controller driver implementation perspective, you still have to
make sure that no client is calling an op on which it is not supposed
to. When you have a mixture like this, a common code would include some
dead paths for certain links.
>
>
>>> Yes, I have 2 types of controllers and I already thought about
>>> multiple controllers in a platform.
>>> What do you expect to do in controller startup/shutdown? Since the
>>> API works on individual links, not the controllers, when would it call
>>> controller_start() and controller_shutdown()? The ipc_link's
>>> startup/shutdown are already non-atomic, if the controller needs any
>>> setup it could do it in the first link's startup and teardown in the
>>> last link's shutdown. Not to forget the resources lke IRQ are usually
>>> per Link which should be acquired/released upon link's use/idle.
>>> While writing the OMAP2 controller driver, did I miss any controller
>>> startup/shutdown ?
>>
>> It just delineates the code better, and has flexibility later on to
>> extend any controller ops or exposing new controller-specific API. The
>> controller start and stop will be called in the same function as
>> ipc_request_channel.
>>
> Already one call ipc_link_ops.startup() reaches the controller upon
> ipc_request_channel.
>
> Do you mean ?
> ipc_con_ops.startup();
> ipc_link_ops.startup();
Yes.
>
> Why have 2 contiguous calls to the controller, to perform similar
> functions? Not to forget the API and clients don't work on IPC
> controllers, but only on links. So it would be out-of-line for them to
> have to worry about the controller.
As I said, mainly to help dilineate the controller implementation code
better. I can live with it not being present, as my startup function
would anyway be refactored into two - one for the controller, and one
for the link itself. And I bet most would do the same if they have
multiple links in a controller.
>
>>
>> Yeah, the pm_runtime_enable cannot be called twice when mailbox is
>> invoked from multiple clients on separate links, so there has to a
>> controller-level logic/ref-counting for that. The clocking for us is on
>> the controller.
>>
> No. You could call pm_runtime_enable/disable any number of times as
> long as they are balanced. The core does refcounting.
Exactly, as long as they are balanced. I have two clients dealing with
two remotes (using two links) so pm_runtime_enable on the h/w block
needs to be called only when the first one comes in. Anyway, this is
just an FYI, as this file will obviously be different since I have to
not break my existing clients.
regards
Suman
^ permalink raw reply [flat|nested] 90+ messages in thread* [RFC 2/3] mailbox: Introduce a new common API
2013-05-09 1:25 ` Suman Anna
@ 2013-05-09 16:35 ` Jassi Brar
-1 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-05-09 16:35 UTC (permalink / raw)
To: linux-arm-kernel
Hi Suman,
On 9 May 2013 06:55, Suman Anna <s-anna@ti.com> wrote:
>> so it can't be driven by the controller. We could make it a Kconfig option.
>> What do you suggest?
>
> I am saying controller/link because they are the ones that knows how the
> physical transport is, and it may vary from one to another. I would
> think that the client dependencies would become a subset of this.
> Kconfig option is fine, but we have to think about the best way of
> representing it from a multi-platform kernel support perspective.
>
Actually it has more to do with clients and less with multiplatform.
Even on single platform the client that needs biggest buffer will
rule. Ditto on multi-platform.
Anyways I get your idea. I'll put a note there to revisit once number
of active clients increase on the platform.
>> As I said the API shouldn't keep internal copies of a client's
>> requests - it will only kill zero-copy use-cases while adding
>> complexity to the core.
>
> This is the reason I was suggesting the nature be dictated by a
> controller, rather than 1 design in core code mandating a certain usage.
> The client & controller together would know the functional integration
> aspect.
>
>> Since it already works fine for OMAP, I request we leave it as such
>> for now at least.
>
> OK, we should make a note of it in some documentation.
>
OK, will do. Thanks.
>>> IMHO. It's calling ipc_links_register twice on the same controller. Idea
>>> was that the API would distinguish different controllers, not the same
>>> one. Also, see the above example if you were to treat a link as Rx or Tx
>>> only (functional behavior differences even though the link is exactly
>>> the same type).
>>>
>> Why is calling ipc_links_register() twice, for a controller with 2
>> classes of links, a problem ?
>
> It will be inelegant, once you start maintaining the list of
> ipc_controllers in the core code. You would have to search all the
> controllers in the list with the matching name, and their links. You
> will need to define multiple controller specific controller structures
> and copy most of the elements from the actual h/w device into the
> containing ipc_controller definition. As you said, the clients deal with
> the links themselves, so making the ops part of the ipc_link definition
> makes it easier on the controller implementations. You are anyway
> caching the ops in ipc_chan (per link) and that is what you are using in
> the core code. Majority of the time, you would be using the same ops for
> all the links, but this gives the flexibility to links. You can avoid
> having multiple controller instance denoting the h/w block, just because
> of the difference in links behavior.
>
I don't see it as an issue, it's just how the code is designed on the
principle that the API works only on links, not controllers. Let us
wait for the driver of such a controller to show up. It should be easy
to change if we see the need.
>>
>> And, No, the API need not know if the link is "RX" or "TX", which is
>> purely a logical function.
>>
>> Please do tell me which controller physically limits its links to do
>> only "RX" or only "TX"? It's just the platform's firmware that
>> dictates who sends and who receives on which link, thereby giving the
>> illusion of a link being a "RX" or "TX" capable only.
>> Even if some h/w limits do exist on links of some controller, still
>> the API shouldn't try to discern a "RX" from a "TX" link. When a
>> client knows which link to open, it will also know if it should read
>> or write to it - this assumption already holds true. If the client
>> writes on a link that it's supposed to listen to remote on, no amount
>> of policing by the API could help.
>>
>> Even if the API tries to differentiate "RX" and "TX" links, the
>> knowledge must come from a client (it's driven by the protocol on the
>> platform), and not the controller because each of its links are
>> physically equally capable to do RX or TX on each end.
>
> The API never differentiates, but the controller implementation has to.
> From a controller driver implementation perspective, you still have to
> make sure that no client is calling an op on which it is not supposed
> to. When you have a mixture like this, a common code would include some
> dead paths for certain links.
>
No, please. The controller driver should not implement any policy (of
allowing/disallowing requests). It should simply try to do as
directed. If the client screwed up even after getting info from
platform_data/DT, let it suffer.
>>>> Yes, I have 2 types of controllers and I already thought about
>>>> multiple controllers in a platform.
>>>> What do you expect to do in controller startup/shutdown? Since the
>>>> API works on individual links, not the controllers, when would it call
>>>> controller_start() and controller_shutdown()? The ipc_link's
>>>> startup/shutdown are already non-atomic, if the controller needs any
>>>> setup it could do it in the first link's startup and teardown in the
>>>> last link's shutdown. Not to forget the resources lke IRQ are usually
>>>> per Link which should be acquired/released upon link's use/idle.
>>>> While writing the OMAP2 controller driver, did I miss any controller
>>>> startup/shutdown ?
>>>
>>> It just delineates the code better, and has flexibility later on to
>>> extend any controller ops or exposing new controller-specific API. The
>>> controller start and stop will be called in the same function as
>>> ipc_request_channel.
>>>
>> Already one call ipc_link_ops.startup() reaches the controller upon
>> ipc_request_channel.
>>
>> Do you mean ?
>> ipc_con_ops.startup();
>> ipc_link_ops.startup();
>
> Yes.
>
And let the controller startup()/shutdown() upon every
link_request/release? If no, then the ipc_con_ops.startup() will be
really executed only upon probe or resume, which is controller
driver's internal matter.
BTW, I have seen my 2 controllers, OMAP, PL320 and read the dbx500
driver. Unsurprisingly none of these have any use for what you call a
special ipc_con_ops.startup(). Lets say if the call were there, what
would the OMAP put in it?
>>> Yeah, the pm_runtime_enable cannot be called twice when mailbox is
>>> invoked from multiple clients on separate links, so there has to a
>>> controller-level logic/ref-counting for that. The clocking for us is on
>>> the controller.
>>>
>> No. You could call pm_runtime_enable/disable any number of times as
>> long as they are balanced. The core does refcounting.
>
> Exactly, as long as they are balanced. I have two clients dealing with
> two remotes (using two links) so pm_runtime_enable on the h/w block
> needs to be called only when the first one comes in.
>
Actually you just gave another reason why the API messing around with
controller's power state is a bad idea.
See how mailbox_startup() tries to balance mbox->ops->startup() and
mailbox_fini() the mbox->ops->shutdown() That's very fragile and the
cause of imbalance between rpm enable/disable, unless your clients are
buggy.
Regards,
-Jassi
^ permalink raw reply [flat|nested] 90+ messages in thread
* Re: [RFC 2/3] mailbox: Introduce a new common API
@ 2013-05-09 16:35 ` Jassi Brar
0 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-05-09 16:35 UTC (permalink / raw)
To: Suman Anna
Cc: Jassi Brar, Loic PALLARDY (loic.pallardy@st.com), Arnd Bergmann,
lkml, linux-arm-kernel@lists.infradead.org, Andy Green
Hi Suman,
On 9 May 2013 06:55, Suman Anna <s-anna@ti.com> wrote:
>> so it can't be driven by the controller. We could make it a Kconfig option.
>> What do you suggest?
>
> I am saying controller/link because they are the ones that knows how the
> physical transport is, and it may vary from one to another. I would
> think that the client dependencies would become a subset of this.
> Kconfig option is fine, but we have to think about the best way of
> representing it from a multi-platform kernel support perspective.
>
Actually it has more to do with clients and less with multiplatform.
Even on single platform the client that needs biggest buffer will
rule. Ditto on multi-platform.
Anyways I get your idea. I'll put a note there to revisit once number
of active clients increase on the platform.
>> As I said the API shouldn't keep internal copies of a client's
>> requests - it will only kill zero-copy use-cases while adding
>> complexity to the core.
>
> This is the reason I was suggesting the nature be dictated by a
> controller, rather than 1 design in core code mandating a certain usage.
> The client & controller together would know the functional integration
> aspect.
>
>> Since it already works fine for OMAP, I request we leave it as such
>> for now at least.
>
> OK, we should make a note of it in some documentation.
>
OK, will do. Thanks.
>>> IMHO. It's calling ipc_links_register twice on the same controller. Idea
>>> was that the API would distinguish different controllers, not the same
>>> one. Also, see the above example if you were to treat a link as Rx or Tx
>>> only (functional behavior differences even though the link is exactly
>>> the same type).
>>>
>> Why is calling ipc_links_register() twice, for a controller with 2
>> classes of links, a problem ?
>
> It will be inelegant, once you start maintaining the list of
> ipc_controllers in the core code. You would have to search all the
> controllers in the list with the matching name, and their links. You
> will need to define multiple controller specific controller structures
> and copy most of the elements from the actual h/w device into the
> containing ipc_controller definition. As you said, the clients deal with
> the links themselves, so making the ops part of the ipc_link definition
> makes it easier on the controller implementations. You are anyway
> caching the ops in ipc_chan (per link) and that is what you are using in
> the core code. Majority of the time, you would be using the same ops for
> all the links, but this gives the flexibility to links. You can avoid
> having multiple controller instance denoting the h/w block, just because
> of the difference in links behavior.
>
I don't see it as an issue, it's just how the code is designed on the
principle that the API works only on links, not controllers. Let us
wait for the driver of such a controller to show up. It should be easy
to change if we see the need.
>>
>> And, No, the API need not know if the link is "RX" or "TX", which is
>> purely a logical function.
>>
>> Please do tell me which controller physically limits its links to do
>> only "RX" or only "TX"? It's just the platform's firmware that
>> dictates who sends and who receives on which link, thereby giving the
>> illusion of a link being a "RX" or "TX" capable only.
>> Even if some h/w limits do exist on links of some controller, still
>> the API shouldn't try to discern a "RX" from a "TX" link. When a
>> client knows which link to open, it will also know if it should read
>> or write to it - this assumption already holds true. If the client
>> writes on a link that it's supposed to listen to remote on, no amount
>> of policing by the API could help.
>>
>> Even if the API tries to differentiate "RX" and "TX" links, the
>> knowledge must come from a client (it's driven by the protocol on the
>> platform), and not the controller because each of its links are
>> physically equally capable to do RX or TX on each end.
>
> The API never differentiates, but the controller implementation has to.
> From a controller driver implementation perspective, you still have to
> make sure that no client is calling an op on which it is not supposed
> to. When you have a mixture like this, a common code would include some
> dead paths for certain links.
>
No, please. The controller driver should not implement any policy (of
allowing/disallowing requests). It should simply try to do as
directed. If the client screwed up even after getting info from
platform_data/DT, let it suffer.
>>>> Yes, I have 2 types of controllers and I already thought about
>>>> multiple controllers in a platform.
>>>> What do you expect to do in controller startup/shutdown? Since the
>>>> API works on individual links, not the controllers, when would it call
>>>> controller_start() and controller_shutdown()? The ipc_link's
>>>> startup/shutdown are already non-atomic, if the controller needs any
>>>> setup it could do it in the first link's startup and teardown in the
>>>> last link's shutdown. Not to forget the resources lke IRQ are usually
>>>> per Link which should be acquired/released upon link's use/idle.
>>>> While writing the OMAP2 controller driver, did I miss any controller
>>>> startup/shutdown ?
>>>
>>> It just delineates the code better, and has flexibility later on to
>>> extend any controller ops or exposing new controller-specific API. The
>>> controller start and stop will be called in the same function as
>>> ipc_request_channel.
>>>
>> Already one call ipc_link_ops.startup() reaches the controller upon
>> ipc_request_channel.
>>
>> Do you mean ?
>> ipc_con_ops.startup();
>> ipc_link_ops.startup();
>
> Yes.
>
And let the controller startup()/shutdown() upon every
link_request/release? If no, then the ipc_con_ops.startup() will be
really executed only upon probe or resume, which is controller
driver's internal matter.
BTW, I have seen my 2 controllers, OMAP, PL320 and read the dbx500
driver. Unsurprisingly none of these have any use for what you call a
special ipc_con_ops.startup(). Lets say if the call were there, what
would the OMAP put in it?
>>> Yeah, the pm_runtime_enable cannot be called twice when mailbox is
>>> invoked from multiple clients on separate links, so there has to a
>>> controller-level logic/ref-counting for that. The clocking for us is on
>>> the controller.
>>>
>> No. You could call pm_runtime_enable/disable any number of times as
>> long as they are balanced. The core does refcounting.
>
> Exactly, as long as they are balanced. I have two clients dealing with
> two remotes (using two links) so pm_runtime_enable on the h/w block
> needs to be called only when the first one comes in.
>
Actually you just gave another reason why the API messing around with
controller's power state is a bad idea.
See how mailbox_startup() tries to balance mbox->ops->startup() and
mailbox_fini() the mbox->ops->shutdown() That's very fragile and the
cause of imbalance between rpm enable/disable, unless your clients are
buggy.
Regards,
-Jassi
^ permalink raw reply [flat|nested] 90+ messages in thread
* [RFC 2/3] mailbox: Introduce a new common API
2013-05-09 16:35 ` Jassi Brar
@ 2013-05-10 0:18 ` Suman Anna
-1 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-05-10 0:18 UTC (permalink / raw)
To: linux-arm-kernel
Hi Jassi,
>
> On 9 May 2013 06:55, Suman Anna <s-anna@ti.com> wrote:
>
>>> so it can't be driven by the controller. We could make it a Kconfig option.
>>> What do you suggest?
>>
>> I am saying controller/link because they are the ones that knows how the
>> physical transport is, and it may vary from one to another. I would
>> think that the client dependencies would become a subset of this.
>> Kconfig option is fine, but we have to think about the best way of
>> representing it from a multi-platform kernel support perspective.
>>
> Actually it has more to do with clients and less with multiplatform.
> Even on single platform the client that needs biggest buffer will
> rule. Ditto on multi-platform.
> Anyways I get your idea. I'll put a note there to revisit once number
> of active clients increase on the platform.
OK thanks.
>>>> IMHO. It's calling ipc_links_register twice on the same controller. Idea
>>>> was that the API would distinguish different controllers, not the same
>>>> one. Also, see the above example if you were to treat a link as Rx or Tx
>>>> only (functional behavior differences even though the link is exactly
>>>> the same type).
>>>>
>>> Why is calling ipc_links_register() twice, for a controller with 2
>>> classes of links, a problem ?
>>
>> It will be inelegant, once you start maintaining the list of
>> ipc_controllers in the core code. You would have to search all the
>> controllers in the list with the matching name, and their links. You
>> will need to define multiple controller specific controller structures
>> and copy most of the elements from the actual h/w device into the
>> containing ipc_controller definition. As you said, the clients deal with
>> the links themselves, so making the ops part of the ipc_link definition
>> makes it easier on the controller implementations. You are anyway
>> caching the ops in ipc_chan (per link) and that is what you are using in
>> the core code. Majority of the time, you would be using the same ops for
>> all the links, but this gives the flexibility to links. You can avoid
>> having multiple controller instance denoting the h/w block, just because
>> of the difference in links behavior.
>>
> I don't see it as an issue, it's just how the code is designed on the
> principle that the API works only on links, not controllers. Let us
> wait for the driver of such a controller to show up. It should be easy
> to change if we see the need.
OK fine, I brought it up since we are defining the API, and defining it
the first time with flexibility eliminates the need later on.
>
>>>
>>> And, No, the API need not know if the link is "RX" or "TX", which is
>>> purely a logical function.
>>>
>>> Please do tell me which controller physically limits its links to do
>>> only "RX" or only "TX"? It's just the platform's firmware that
>>> dictates who sends and who receives on which link, thereby giving the
>>> illusion of a link being a "RX" or "TX" capable only.
>>> Even if some h/w limits do exist on links of some controller, still
>>> the API shouldn't try to discern a "RX" from a "TX" link. When a
>>> client knows which link to open, it will also know if it should read
>>> or write to it - this assumption already holds true. If the client
>>> writes on a link that it's supposed to listen to remote on, no amount
>>> of policing by the API could help.
>>>
>>> Even if the API tries to differentiate "RX" and "TX" links, the
>>> knowledge must come from a client (it's driven by the protocol on the
>>> platform), and not the controller because each of its links are
>>> physically equally capable to do RX or TX on each end.
>>
>> The API never differentiates, but the controller implementation has to.
>> From a controller driver implementation perspective, you still have to
>> make sure that no client is calling an op on which it is not supposed
>> to. When you have a mixture like this, a common code would include some
>> dead paths for certain links.
>>
> No, please. The controller driver should not implement any policy (of
> allowing/disallowing requests). It should simply try to do as
> directed. If the client screwed up even after getting info from
> platform_data/DT, let it suffer.
This would be the same as a client trying to misconfigure a link, you
cannot blame on a client screw up. The controller driver has to take
care of what it ought to check for in the startup.
>
>>>>> Yes, I have 2 types of controllers and I already thought about
>>>>> multiple controllers in a platform.
>>>>> What do you expect to do in controller startup/shutdown? Since the
>>>>> API works on individual links, not the controllers, when would it call
>>>>> controller_start() and controller_shutdown()? The ipc_link's
>>>>> startup/shutdown are already non-atomic, if the controller needs any
>>>>> setup it could do it in the first link's startup and teardown in the
>>>>> last link's shutdown. Not to forget the resources lke IRQ are usually
>>>>> per Link which should be acquired/released upon link's use/idle.
>>>>> While writing the OMAP2 controller driver, did I miss any controller
>>>>> startup/shutdown ?
>>>>
>>>> It just delineates the code better, and has flexibility later on to
>>>> extend any controller ops or exposing new controller-specific API. The
>>>> controller start and stop will be called in the same function as
>>>> ipc_request_channel.
>>>>
>>> Already one call ipc_link_ops.startup() reaches the controller upon
>>> ipc_request_channel.
>>>
>>> Do you mean ?
>>> ipc_con_ops.startup();
>>> ipc_link_ops.startup();
>>
>> Yes.
>>
> And let the controller startup()/shutdown() upon every
> link_request/release?
Yes, and the controller driver can take care of any ref-counting or
whatever other logic it needs to maintain. You see this outside in the
previous mailbox code, but that is not the responsibility of the core code.
>
> BTW, I have seen my 2 controllers, OMAP, PL320 and read the dbx500
> driver. Unsurprisingly none of these have any use for what you call a
> special ipc_con_ops.startup(). Lets say if the call were there, what
> would the OMAP put in it?
Enabling the clock for the device. The clock is for the entire IP, a
link has no power/clock dependencies.
>
>>>> Yeah, the pm_runtime_enable cannot be called twice when mailbox is
>>>> invoked from multiple clients on separate links, so there has to a
>>>> controller-level logic/ref-counting for that. The clocking for us is on
>>>> the controller.
>>>>
>>> No. You could call pm_runtime_enable/disable any number of times as
>>> long as they are balanced. The core does refcounting.
>>
>> Exactly, as long as they are balanced. I have two clients dealing with
>> two remotes (using two links) so pm_runtime_enable on the h/w block
>> needs to be called only when the first one comes in.
>>
> Actually you just gave another reason why the API messing around with
> controller's power state is a bad idea.
Where do you expect to power up the device (obviously this depends on
the SoC, and its functional purpose)?
> See how mailbox_startup() tries to balance mbox->ops->startup() and
> mailbox_fini() the mbox->ops->shutdown() That's very fragile and the
> cause of imbalance between rpm enable/disable, unless your clients are
> buggy.
Yeah, it is kinda messed up in the existing code, the startup defined
there is really for the controller and not the link, and that's why you
see all the ref-counting balancing logic. The rpm enable/disable being
called is on the controller's dev, not the link's dev - maybe that's
what confused you.
regards
Suman
^ permalink raw reply [flat|nested] 90+ messages in thread
* Re: [RFC 2/3] mailbox: Introduce a new common API
@ 2013-05-10 0:18 ` Suman Anna
0 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-05-10 0:18 UTC (permalink / raw)
To: Jassi Brar
Cc: Jassi Brar, Loic PALLARDY (loic.pallardy@st.com), Arnd Bergmann,
lkml, linux-arm-kernel@lists.infradead.org, Andy Green
Hi Jassi,
>
> On 9 May 2013 06:55, Suman Anna <s-anna@ti.com> wrote:
>
>>> so it can't be driven by the controller. We could make it a Kconfig option.
>>> What do you suggest?
>>
>> I am saying controller/link because they are the ones that knows how the
>> physical transport is, and it may vary from one to another. I would
>> think that the client dependencies would become a subset of this.
>> Kconfig option is fine, but we have to think about the best way of
>> representing it from a multi-platform kernel support perspective.
>>
> Actually it has more to do with clients and less with multiplatform.
> Even on single platform the client that needs biggest buffer will
> rule. Ditto on multi-platform.
> Anyways I get your idea. I'll put a note there to revisit once number
> of active clients increase on the platform.
OK thanks.
>>>> IMHO. It's calling ipc_links_register twice on the same controller. Idea
>>>> was that the API would distinguish different controllers, not the same
>>>> one. Also, see the above example if you were to treat a link as Rx or Tx
>>>> only (functional behavior differences even though the link is exactly
>>>> the same type).
>>>>
>>> Why is calling ipc_links_register() twice, for a controller with 2
>>> classes of links, a problem ?
>>
>> It will be inelegant, once you start maintaining the list of
>> ipc_controllers in the core code. You would have to search all the
>> controllers in the list with the matching name, and their links. You
>> will need to define multiple controller specific controller structures
>> and copy most of the elements from the actual h/w device into the
>> containing ipc_controller definition. As you said, the clients deal with
>> the links themselves, so making the ops part of the ipc_link definition
>> makes it easier on the controller implementations. You are anyway
>> caching the ops in ipc_chan (per link) and that is what you are using in
>> the core code. Majority of the time, you would be using the same ops for
>> all the links, but this gives the flexibility to links. You can avoid
>> having multiple controller instance denoting the h/w block, just because
>> of the difference in links behavior.
>>
> I don't see it as an issue, it's just how the code is designed on the
> principle that the API works only on links, not controllers. Let us
> wait for the driver of such a controller to show up. It should be easy
> to change if we see the need.
OK fine, I brought it up since we are defining the API, and defining it
the first time with flexibility eliminates the need later on.
>
>>>
>>> And, No, the API need not know if the link is "RX" or "TX", which is
>>> purely a logical function.
>>>
>>> Please do tell me which controller physically limits its links to do
>>> only "RX" or only "TX"? It's just the platform's firmware that
>>> dictates who sends and who receives on which link, thereby giving the
>>> illusion of a link being a "RX" or "TX" capable only.
>>> Even if some h/w limits do exist on links of some controller, still
>>> the API shouldn't try to discern a "RX" from a "TX" link. When a
>>> client knows which link to open, it will also know if it should read
>>> or write to it - this assumption already holds true. If the client
>>> writes on a link that it's supposed to listen to remote on, no amount
>>> of policing by the API could help.
>>>
>>> Even if the API tries to differentiate "RX" and "TX" links, the
>>> knowledge must come from a client (it's driven by the protocol on the
>>> platform), and not the controller because each of its links are
>>> physically equally capable to do RX or TX on each end.
>>
>> The API never differentiates, but the controller implementation has to.
>> From a controller driver implementation perspective, you still have to
>> make sure that no client is calling an op on which it is not supposed
>> to. When you have a mixture like this, a common code would include some
>> dead paths for certain links.
>>
> No, please. The controller driver should not implement any policy (of
> allowing/disallowing requests). It should simply try to do as
> directed. If the client screwed up even after getting info from
> platform_data/DT, let it suffer.
This would be the same as a client trying to misconfigure a link, you
cannot blame on a client screw up. The controller driver has to take
care of what it ought to check for in the startup.
>
>>>>> Yes, I have 2 types of controllers and I already thought about
>>>>> multiple controllers in a platform.
>>>>> What do you expect to do in controller startup/shutdown? Since the
>>>>> API works on individual links, not the controllers, when would it call
>>>>> controller_start() and controller_shutdown()? The ipc_link's
>>>>> startup/shutdown are already non-atomic, if the controller needs any
>>>>> setup it could do it in the first link's startup and teardown in the
>>>>> last link's shutdown. Not to forget the resources lke IRQ are usually
>>>>> per Link which should be acquired/released upon link's use/idle.
>>>>> While writing the OMAP2 controller driver, did I miss any controller
>>>>> startup/shutdown ?
>>>>
>>>> It just delineates the code better, and has flexibility later on to
>>>> extend any controller ops or exposing new controller-specific API. The
>>>> controller start and stop will be called in the same function as
>>>> ipc_request_channel.
>>>>
>>> Already one call ipc_link_ops.startup() reaches the controller upon
>>> ipc_request_channel.
>>>
>>> Do you mean ?
>>> ipc_con_ops.startup();
>>> ipc_link_ops.startup();
>>
>> Yes.
>>
> And let the controller startup()/shutdown() upon every
> link_request/release?
Yes, and the controller driver can take care of any ref-counting or
whatever other logic it needs to maintain. You see this outside in the
previous mailbox code, but that is not the responsibility of the core code.
>
> BTW, I have seen my 2 controllers, OMAP, PL320 and read the dbx500
> driver. Unsurprisingly none of these have any use for what you call a
> special ipc_con_ops.startup(). Lets say if the call were there, what
> would the OMAP put in it?
Enabling the clock for the device. The clock is for the entire IP, a
link has no power/clock dependencies.
>
>>>> Yeah, the pm_runtime_enable cannot be called twice when mailbox is
>>>> invoked from multiple clients on separate links, so there has to a
>>>> controller-level logic/ref-counting for that. The clocking for us is on
>>>> the controller.
>>>>
>>> No. You could call pm_runtime_enable/disable any number of times as
>>> long as they are balanced. The core does refcounting.
>>
>> Exactly, as long as they are balanced. I have two clients dealing with
>> two remotes (using two links) so pm_runtime_enable on the h/w block
>> needs to be called only when the first one comes in.
>>
> Actually you just gave another reason why the API messing around with
> controller's power state is a bad idea.
Where do you expect to power up the device (obviously this depends on
the SoC, and its functional purpose)?
> See how mailbox_startup() tries to balance mbox->ops->startup() and
> mailbox_fini() the mbox->ops->shutdown() That's very fragile and the
> cause of imbalance between rpm enable/disable, unless your clients are
> buggy.
Yeah, it is kinda messed up in the existing code, the startup defined
there is really for the controller and not the link, and that's why you
see all the ref-counting balancing logic. The rpm enable/disable being
called is on the controller's dev, not the link's dev - maybe that's
what confused you.
regards
Suman
^ permalink raw reply [flat|nested] 90+ messages in thread
* [RFC 2/3] mailbox: Introduce a new common API
2013-05-10 0:18 ` Suman Anna
@ 2013-05-10 10:06 ` Jassi Brar
-1 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-05-10 10:06 UTC (permalink / raw)
To: linux-arm-kernel
Hello Suman,
On 10 May 2013 05:48, Suman Anna <s-anna@ti.com> wrote:
>> No, please. The controller driver should not implement any policy (of
>> allowing/disallowing requests). It should simply try to do as
>> directed. If the client screwed up even after getting info from
>> platform_data/DT, let it suffer.
>
> This would be the same as a client trying to misconfigure a link, you
> cannot blame on a client screw up.
>
One kernel code should not suspect other kernel code's intentions. The
safety/security is in allowing only well written and trusted code
inside the kernel space.
If a client misconfigures a link, the best we could do is call it
buggy. BTW such a hell-bent client would rather screw up with a BUG(),
what do we do then?
So no please, I am not convinced the controller driver should have
anything to do with allow/disallow policy, or any policy at all.
>>>>>
>>>> Already one call ipc_link_ops.startup() reaches the controller upon
>>>> ipc_request_channel.
>>>>
>>>> Do you mean ?
>>>> ipc_con_ops.startup();
>>>> ipc_link_ops.startup();
>>>
>>> Yes.
>>>
>> And let the controller startup()/shutdown() upon every
>> link_request/release?
>
> Yes, and the controller driver can take care of any ref-counting or
> whatever other logic it needs to maintain. You see this outside in the
> previous mailbox code, but that is not the responsibility of the core code.
>
The pm_runtime imbalance that you mentioned is the result of OMAP's
API trying to do just that.
>>
>> BTW, I have seen my 2 controllers, OMAP, PL320 and read the dbx500
>> driver. Unsurprisingly none of these have any use for what you call a
>> special ipc_con_ops.startup(). Lets say if the call were there, what
>> would the OMAP put in it?
>
> Enabling the clock for the device. The clock is for the entire IP, a
> link has no power/clock dependencies.
>
OMAP controller driver doesn't do clocks.
It does do rpm though, which does a better job of what OMAP needs.
Please see how pm_runtime_xxx calls are made without worry at each
ipc_link_ops.startup/shutdown.
For controllers that do do clocks, having contiguous
ipc_con_ops.startup();
ipc_link_ops.startup();
doesn't save them any complexity. They still have to keep track of
first ipc_link_ops.startup() call in ipc_con_ops.startup() and last
ipc_link_ops.shutdown() call in ipc_con_ops.shutdown()
Which they could very well do in link_ops, if not rpm.
>>
>>>>> Yeah, the pm_runtime_enable cannot be called twice when mailbox is
>>>>> invoked from multiple clients on separate links, so there has to a
>>>>> controller-level logic/ref-counting for that. The clocking for us is on
>>>>> the controller.
>>>>>
>>>> No. You could call pm_runtime_enable/disable any number of times as
>>>> long as they are balanced. The core does refcounting.
>>>
>>> Exactly, as long as they are balanced. I have two clients dealing with
>>> two remotes (using two links) so pm_runtime_enable on the h/w block
>>> needs to be called only when the first one comes in.
>>>
>> Actually you just gave another reason why the API messing around with
>> controller's power state is a bad idea.
>
> Where do you expect to power up the device (obviously this depends on
> the SoC, and its functional purpose)?
>
Use RPM. Just like OMAP controller driver already does -
unconditionally call pm_runtime_get upon each link request and
pm_runtime_put upon each link release. Neither the controller nor the
API would have to do refcounting. If rpm isn't supported, only then
the controller should do refcounting within linp_ops.startup/shutdown
(anyways you want the con_ops and link_ops to be always contiguous).
>> See how mailbox_startup() tries to balance mbox->ops->startup() and
>> mailbox_fini() the mbox->ops->shutdown() That's very fragile and the
>> cause of imbalance between rpm enable/disable, unless your clients are
>> buggy.
>
> Yeah, it is kinda messed up in the existing code, the startup defined
> there is really for the controller and not the link, and that's why you
> see all the ref-counting balancing logic. The rpm enable/disable being
> called is on the controller's dev, not the link's dev - maybe that's
> what confused you.
>
I wrote the driver for this api, so I do realize the rpm is on
controller. Links are not devices anyways.
cheers,
jassi
^ permalink raw reply [flat|nested] 90+ messages in thread* Re: [RFC 2/3] mailbox: Introduce a new common API
@ 2013-05-10 10:06 ` Jassi Brar
0 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-05-10 10:06 UTC (permalink / raw)
To: Suman Anna
Cc: Jassi Brar, Loic PALLARDY (loic.pallardy@st.com), Arnd Bergmann,
lkml, linux-arm-kernel@lists.infradead.org, Andy Green
Hello Suman,
On 10 May 2013 05:48, Suman Anna <s-anna@ti.com> wrote:
>> No, please. The controller driver should not implement any policy (of
>> allowing/disallowing requests). It should simply try to do as
>> directed. If the client screwed up even after getting info from
>> platform_data/DT, let it suffer.
>
> This would be the same as a client trying to misconfigure a link, you
> cannot blame on a client screw up.
>
One kernel code should not suspect other kernel code's intentions. The
safety/security is in allowing only well written and trusted code
inside the kernel space.
If a client misconfigures a link, the best we could do is call it
buggy. BTW such a hell-bent client would rather screw up with a BUG(),
what do we do then?
So no please, I am not convinced the controller driver should have
anything to do with allow/disallow policy, or any policy at all.
>>>>>
>>>> Already one call ipc_link_ops.startup() reaches the controller upon
>>>> ipc_request_channel.
>>>>
>>>> Do you mean ?
>>>> ipc_con_ops.startup();
>>>> ipc_link_ops.startup();
>>>
>>> Yes.
>>>
>> And let the controller startup()/shutdown() upon every
>> link_request/release?
>
> Yes, and the controller driver can take care of any ref-counting or
> whatever other logic it needs to maintain. You see this outside in the
> previous mailbox code, but that is not the responsibility of the core code.
>
The pm_runtime imbalance that you mentioned is the result of OMAP's
API trying to do just that.
>>
>> BTW, I have seen my 2 controllers, OMAP, PL320 and read the dbx500
>> driver. Unsurprisingly none of these have any use for what you call a
>> special ipc_con_ops.startup(). Lets say if the call were there, what
>> would the OMAP put in it?
>
> Enabling the clock for the device. The clock is for the entire IP, a
> link has no power/clock dependencies.
>
OMAP controller driver doesn't do clocks.
It does do rpm though, which does a better job of what OMAP needs.
Please see how pm_runtime_xxx calls are made without worry at each
ipc_link_ops.startup/shutdown.
For controllers that do do clocks, having contiguous
ipc_con_ops.startup();
ipc_link_ops.startup();
doesn't save them any complexity. They still have to keep track of
first ipc_link_ops.startup() call in ipc_con_ops.startup() and last
ipc_link_ops.shutdown() call in ipc_con_ops.shutdown()
Which they could very well do in link_ops, if not rpm.
>>
>>>>> Yeah, the pm_runtime_enable cannot be called twice when mailbox is
>>>>> invoked from multiple clients on separate links, so there has to a
>>>>> controller-level logic/ref-counting for that. The clocking for us is on
>>>>> the controller.
>>>>>
>>>> No. You could call pm_runtime_enable/disable any number of times as
>>>> long as they are balanced. The core does refcounting.
>>>
>>> Exactly, as long as they are balanced. I have two clients dealing with
>>> two remotes (using two links) so pm_runtime_enable on the h/w block
>>> needs to be called only when the first one comes in.
>>>
>> Actually you just gave another reason why the API messing around with
>> controller's power state is a bad idea.
>
> Where do you expect to power up the device (obviously this depends on
> the SoC, and its functional purpose)?
>
Use RPM. Just like OMAP controller driver already does -
unconditionally call pm_runtime_get upon each link request and
pm_runtime_put upon each link release. Neither the controller nor the
API would have to do refcounting. If rpm isn't supported, only then
the controller should do refcounting within linp_ops.startup/shutdown
(anyways you want the con_ops and link_ops to be always contiguous).
>> See how mailbox_startup() tries to balance mbox->ops->startup() and
>> mailbox_fini() the mbox->ops->shutdown() That's very fragile and the
>> cause of imbalance between rpm enable/disable, unless your clients are
>> buggy.
>
> Yeah, it is kinda messed up in the existing code, the startup defined
> there is really for the controller and not the link, and that's why you
> see all the ref-counting balancing logic. The rpm enable/disable being
> called is on the controller's dev, not the link's dev - maybe that's
> what confused you.
>
I wrote the driver for this api, so I do realize the rpm is on
controller. Links are not devices anyways.
cheers,
jassi
^ permalink raw reply [flat|nested] 90+ messages in thread* [RFC 2/3] mailbox: Introduce a new common API
2013-05-10 10:06 ` Jassi Brar
@ 2013-05-10 16:41 ` Suman Anna
-1 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-05-10 16:41 UTC (permalink / raw)
To: linux-arm-kernel
Jassi,
>>> See how mailbox_startup() tries to balance mbox->ops->startup() and
>>> mailbox_fini() the mbox->ops->shutdown() That's very fragile and the
>>> cause of imbalance between rpm enable/disable, unless your clients are
>>> buggy.
>>
>> Yeah, it is kinda messed up in the existing code, the startup defined
>> there is really for the controller and not the link, and that's why you
>> see all the ref-counting balancing logic. The rpm enable/disable being
>> called is on the controller's dev, not the link's dev - maybe that's
>> what confused you.
>>
> I wrote the driver for this api, so I do realize the rpm is on
> controller. Links are not devices anyways.
>
You might be missing the minor detail. The pm_runtime_enable is in
startup, which is called for every link. So, I get an unbalanced
warning, when I have
ipc_request_channel(client1); -> link1.startup
ipc_request_channel(clinet2); -> link2.startup
Anyway, it is a non-issue since I understand the context and the fix.
regards
Suman
^ permalink raw reply [flat|nested] 90+ messages in thread
* Re: [RFC 2/3] mailbox: Introduce a new common API
@ 2013-05-10 16:41 ` Suman Anna
0 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-05-10 16:41 UTC (permalink / raw)
To: Jassi Brar
Cc: Jassi Brar, Loic PALLARDY (loic.pallardy@st.com), Arnd Bergmann,
lkml, linux-arm-kernel@lists.infradead.org, Andy Green
Jassi,
>>> See how mailbox_startup() tries to balance mbox->ops->startup() and
>>> mailbox_fini() the mbox->ops->shutdown() That's very fragile and the
>>> cause of imbalance between rpm enable/disable, unless your clients are
>>> buggy.
>>
>> Yeah, it is kinda messed up in the existing code, the startup defined
>> there is really for the controller and not the link, and that's why you
>> see all the ref-counting balancing logic. The rpm enable/disable being
>> called is on the controller's dev, not the link's dev - maybe that's
>> what confused you.
>>
> I wrote the driver for this api, so I do realize the rpm is on
> controller. Links are not devices anyways.
>
You might be missing the minor detail. The pm_runtime_enable is in
startup, which is called for every link. So, I get an unbalanced
warning, when I have
ipc_request_channel(client1); -> link1.startup
ipc_request_channel(clinet2); -> link2.startup
Anyway, it is a non-issue since I understand the context and the fix.
regards
Suman
^ permalink raw reply [flat|nested] 90+ messages in thread
* [RFC 3/3] mailbox: pl320: Introduce common API driver
2013-04-27 4:51 ` Jassi Brar
` (2 preceding siblings ...)
(?)
@ 2013-04-27 18:14 ` jassisinghbrar
2013-04-29 16:44 ` Suman Anna
-1 siblings, 1 reply; 90+ messages in thread
From: jassisinghbrar @ 2013-04-27 18:14 UTC (permalink / raw)
To: s-anna, loic.pallardy
Cc: arnd, linux-kernel, linux-arm-kernel, andy.green, Jassi Brar
From: Jassi Brar <jaswinder.singh@linaro.org>
Convert the PL320 controller driver to work with the common
mailbox API. Also convert the only user of PL320, highbank-cpufreq.c
to work with thee API. Drop the obsoleted driver pl320-ipc.c
Signed-off-by: Jassi Brar <jaswinder.singh@linaro.org>
---
drivers/cpufreq/highbank-cpufreq.c | 22 +++-
drivers/mailbox/Makefile | 2 +-
drivers/mailbox/{pl320-ipc.c => pl320.c} | 194 ++++++++++++++++--------------
include/linux/pl320-ipc.h | 17 ---
4 files changed, 125 insertions(+), 110 deletions(-)
rename drivers/mailbox/{pl320-ipc.c => pl320.c} (51%)
delete mode 100644 include/linux/pl320-ipc.h
diff --git a/drivers/cpufreq/highbank-cpufreq.c b/drivers/cpufreq/highbank-cpufreq.c
index 3118b87..5c057e0 100644
--- a/drivers/cpufreq/highbank-cpufreq.c
+++ b/drivers/cpufreq/highbank-cpufreq.c
@@ -19,7 +19,7 @@
#include <linux/cpu.h>
#include <linux/err.h>
#include <linux/of.h>
-#include <linux/pl320-ipc.h>
+#include <linux/mailbox_client.h>
#include <linux/platform_device.h>
#define HB_CPUFREQ_CHANGE_NOTE 0x80000001
@@ -29,8 +29,26 @@
static int hb_voltage_change(unsigned int freq)
{
u32 msg[HB_CPUFREQ_IPC_LEN] = {HB_CPUFREQ_CHANGE_NOTE, freq / 1000000};
+ struct ipc_client cl;
+ int ret = -ETIMEDOUT;
+ void *chan;
- return pl320_ipc_transmit(msg);
+ cl.rxcb = NULL;
+ cl.txcb = NULL;
+ cl.tx_block = true;
+ cl.tx_tout = 1000; /* 1 sec */
+ cl.cntlr_data = NULL;
+ cl.knows_txdone = false;
+ cl.chan_name = "pl320:A9_to_M3";
+
+ chan = ipc_request_channel(&cl);
+
+ if (ipc_send_message(chan, (void *)msg))
+ ret = msg[1]; /* PL320 updates buffer with FIFO after ACK */
+
+ ipc_free_channel(chan);
+
+ return ret;
}
static int hb_cpufreq_clk_notify(struct notifier_block *nb,
diff --git a/drivers/mailbox/Makefile b/drivers/mailbox/Makefile
index fefef7e..7b897f3 100644
--- a/drivers/mailbox/Makefile
+++ b/drivers/mailbox/Makefile
@@ -2,4 +2,4 @@
obj-$(CONFIG_MAILBOX) += mailbox.o
-obj-$(CONFIG_PL320_MBOX) += pl320-ipc.o
+obj-$(CONFIG_PL320_MBOX) += pl320.o
diff --git a/drivers/mailbox/pl320-ipc.c b/drivers/mailbox/pl320.c
similarity index 51%
rename from drivers/mailbox/pl320-ipc.c
rename to drivers/mailbox/pl320.c
index f3755e0..98b40f4 100644
--- a/drivers/mailbox/pl320-ipc.c
+++ b/drivers/mailbox/pl320.c
@@ -25,8 +25,9 @@
#include <linux/spinlock.h>
#include <linux/device.h>
#include <linux/amba/bus.h>
-
-#include <linux/pl320-ipc.h>
+#include <linux/slab.h>
+#include <linux/platform_device.h>
+#include <linux/mailbox_controller.h>
#define IPCMxSOURCE(m) ((m) * 0x40)
#define IPCMxDSET(m) (((m) * 0x40) + 0x004)
@@ -44,114 +45,86 @@
#define MBOX_MASK(n) (1 << (n))
#define IPC_TX_MBOX 1
-#define IPC_RX_MBOX 2
#define CHAN_MASK(n) (1 << (n))
#define A9_SOURCE 1
#define M3_SOURCE 0
-static void __iomem *ipc_base;
-static int ipc_irq;
-static DEFINE_MUTEX(ipc_m1_lock);
-static DECLARE_COMPLETION(ipc_completion);
-static ATOMIC_NOTIFIER_HEAD(ipc_notifier);
+struct pl320_con {
+ u32 *data;
+ int ipc_irq;
+ struct device *dev;
+ struct ipc_link link;
+ void __iomem *ipc_base;
+ struct ipc_controller ipc_con;
+};
-static inline void set_destination(int source, int mbox)
+static inline struct pl320_con *to_pl320(struct ipc_link *l)
{
- __raw_writel(CHAN_MASK(source), ipc_base + IPCMxDSET(mbox));
- __raw_writel(CHAN_MASK(source), ipc_base + IPCMxMSET(mbox));
-}
+ if (!l)
+ return NULL;
-static inline void clear_destination(int source, int mbox)
-{
- __raw_writel(CHAN_MASK(source), ipc_base + IPCMxDCLEAR(mbox));
- __raw_writel(CHAN_MASK(source), ipc_base + IPCMxMCLEAR(mbox));
+ return container_of(l, struct pl320_con, link);
}
-static void __ipc_send(int mbox, u32 *data)
+static irqreturn_t ipc_handler(int irq, void *p)
{
- int i;
- for (i = 0; i < 7; i++)
- __raw_writel(data[i], ipc_base + IPCMxDR(mbox, i));
- __raw_writel(0x1, ipc_base + IPCMxSEND(mbox));
-}
+ struct ipc_link *link = (struct ipc_link *)p;
+ struct pl320_con *pl320 = to_pl320(link);
+ void __iomem *ipc_base = pl320->ipc_base;
+ u32 irq_stat;
-static u32 __ipc_rcv(int mbox, u32 *data)
-{
- int i;
- for (i = 0; i < 7; i++)
- data[i] = __raw_readl(ipc_base + IPCMxDR(mbox, i));
- return data[1];
-}
+ irq_stat = __raw_readl(ipc_base + IPCMMIS(1));
+ if (irq_stat & MBOX_MASK(IPC_TX_MBOX)) {
+ u32 *data = pl320->data;
+ int i;
-/* blocking implmentation from the A9 side, not usuable in interrupts! */
-int pl320_ipc_transmit(u32 *data)
-{
- int ret;
+ __raw_writel(0, ipc_base + IPCMxSEND(IPC_TX_MBOX));
+
+ /*
+ * The PL320 driver specifies that the send buffer
+ * will be overwritten by same fifo upon TX ACK.
+ */
+ for (i = 0; i < 7; i++)
+ data[i] = __raw_readl(ipc_base
+ + IPCMxDR(IPC_TX_MBOX, i));
- mutex_lock(&ipc_m1_lock);
+ ipc_link_txdone(link, XFER_OK);
- init_completion(&ipc_completion);
- __ipc_send(IPC_TX_MBOX, data);
- ret = wait_for_completion_timeout(&ipc_completion,
- msecs_to_jiffies(1000));
- if (ret == 0) {
- ret = -ETIMEDOUT;
- goto out;
+ pl320->data = NULL;
}
- ret = __ipc_rcv(IPC_TX_MBOX, data);
-out:
- mutex_unlock(&ipc_m1_lock);
- return ret;
+ return IRQ_HANDLED;
}
-EXPORT_SYMBOL_GPL(pl320_ipc_transmit);
-static irqreturn_t ipc_handler(int irq, void *dev)
+static int pl320_send_data(struct ipc_link *link, void *msg)
{
- u32 irq_stat;
- u32 data[7];
+ struct pl320_con *pl320 = to_pl320(link);
+ void __iomem *ipc_base = pl320->ipc_base;
+ u32 *data = (u32 *)msg;
+ int i;
- irq_stat = __raw_readl(ipc_base + IPCMMIS(1));
- if (irq_stat & MBOX_MASK(IPC_TX_MBOX)) {
- __raw_writel(0, ipc_base + IPCMxSEND(IPC_TX_MBOX));
- complete(&ipc_completion);
- }
- if (irq_stat & MBOX_MASK(IPC_RX_MBOX)) {
- __ipc_rcv(IPC_RX_MBOX, data);
- atomic_notifier_call_chain(&ipc_notifier, data[0], data + 1);
- __raw_writel(2, ipc_base + IPCMxSEND(IPC_RX_MBOX));
- }
+ pl320->data = data;
- return IRQ_HANDLED;
-}
+ for (i = 0; i < 7; i++)
+ __raw_writel(data[i], ipc_base + IPCMxDR(IPC_TX_MBOX, i));
-int pl320_ipc_register_notifier(struct notifier_block *nb)
-{
- return atomic_notifier_chain_register(&ipc_notifier, nb);
-}
-EXPORT_SYMBOL_GPL(pl320_ipc_register_notifier);
+ __raw_writel(0x1, ipc_base + IPCMxSEND(IPC_TX_MBOX));
-int pl320_ipc_unregister_notifier(struct notifier_block *nb)
-{
- return atomic_notifier_chain_unregister(&ipc_notifier, nb);
+ return 0;
}
-EXPORT_SYMBOL_GPL(pl320_ipc_unregister_notifier);
-static int pl320_probe(struct amba_device *adev, const struct amba_id *id)
+static int pl320_startup(struct ipc_link *link, void *ignored)
{
- int ret;
-
- ipc_base = ioremap(adev->res.start, resource_size(&adev->res));
- if (ipc_base == NULL)
- return -ENOMEM;
+ struct pl320_con *pl320 = to_pl320(link);
+ void __iomem *ipc_base = pl320->ipc_base;
+ int err, ipc_irq = pl320->ipc_irq;
__raw_writel(0, ipc_base + IPCMxSEND(IPC_TX_MBOX));
- ipc_irq = adev->irq[0];
- ret = request_irq(ipc_irq, ipc_handler, 0, dev_name(&adev->dev), NULL);
- if (ret < 0)
- goto err;
+ err = request_irq(ipc_irq, ipc_handler, 0, dev_name(pl320->dev), link);
+ if (err)
+ return err;
/* Init slow mailbox */
__raw_writel(CHAN_MASK(A9_SOURCE),
@@ -161,17 +134,58 @@ static int pl320_probe(struct amba_device *adev, const struct amba_id *id)
__raw_writel(CHAN_MASK(M3_SOURCE) | CHAN_MASK(A9_SOURCE),
ipc_base + IPCMxMSET(IPC_TX_MBOX));
- /* Init receive mailbox */
- __raw_writel(CHAN_MASK(M3_SOURCE),
- ipc_base + IPCMxSOURCE(IPC_RX_MBOX));
- __raw_writel(CHAN_MASK(A9_SOURCE),
- ipc_base + IPCMxDSET(IPC_RX_MBOX));
- __raw_writel(CHAN_MASK(M3_SOURCE) | CHAN_MASK(A9_SOURCE),
- ipc_base + IPCMxMSET(IPC_RX_MBOX));
-
+ pl320->data = NULL;
return 0;
-err:
- iounmap(ipc_base);
+}
+
+static void pl320_shutdown(struct ipc_link *link)
+{
+ struct pl320_con *pl320 = to_pl320(link);
+
+ pl320->data = NULL;
+ free_irq(pl320->ipc_irq, link);
+}
+
+static struct ipc_link_ops pl320_ops = {
+ .send_data = pl320_send_data,
+ .startup = pl320_startup,
+ .shutdown = pl320_shutdown,
+};
+
+static int pl320_probe(struct amba_device *adev, const struct amba_id *id)
+{
+ struct pl320_con *pl320;
+ struct ipc_link *l[2];
+ int ret;
+
+ pl320 = kzalloc(sizeof(struct pl320_con), GFP_KERNEL);
+ if (!pl320)
+ return -ENOMEM;
+
+ pl320->ipc_base = ioremap(adev->res.start, resource_size(&adev->res));
+ if (pl320->ipc_base == NULL) {
+ kfree(pl320);
+ return -ENOMEM;
+ }
+
+ pl320->dev = &adev->dev;
+ pl320->ipc_irq = adev->irq[0];
+ amba_set_drvdata(adev, pl320);
+
+ l[0] = &pl320->link;
+ l[1] = NULL;
+ pl320->ipc_con.links = l;
+ pl320->ipc_con.txdone_irq = true;
+ pl320->ipc_con.ops = &pl320_ops;
+ snprintf(pl320->link.link_name, 16, "A9_to_M3");
+ snprintf(pl320->ipc_con.controller_name, 16, "pl320");
+
+ ret = ipc_links_register(&pl320->ipc_con);
+ if (ret) {
+ iounmap(pl320->ipc_base);
+ kfree(pl320);
+ }
+
return ret;
}
diff --git a/include/linux/pl320-ipc.h b/include/linux/pl320-ipc.h
deleted file mode 100644
index 5161f63..0000000
--- a/include/linux/pl320-ipc.h
+++ /dev/null
@@ -1,17 +0,0 @@
-/*
- * This program is free software; you can redistribute it and/or modify it
- * under the terms and conditions of the GNU General Public License,
- * version 2, as published by the Free Software Foundation.
- *
- * This program is distributed in the hope it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
- * more details.
- *
- * You should have received a copy of the GNU General Public License along with
- * this program. If not, see <http://www.gnu.org/licenses/>.
- */
-
-int pl320_ipc_transmit(u32 *data);
-int pl320_ipc_register_notifier(struct notifier_block *nb);
-int pl320_ipc_unregister_notifier(struct notifier_block *nb);
--
1.7.10.4
^ permalink raw reply related [flat|nested] 90+ messages in thread* [RFC 3/3] mailbox: pl320: Introduce common API driver
2013-04-27 18:14 ` [RFC 3/3] mailbox: pl320: Introduce common API driver jassisinghbrar
@ 2013-04-29 16:44 ` Suman Anna
0 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-04-29 16:44 UTC (permalink / raw)
To: linux-arm-kernel
On 04/27/2013 01:14 PM, jassisinghbrar at gmail.com wrote:
> From: Jassi Brar <jaswinder.singh@linaro.org>
>
> Convert the PL320 controller driver to work with the common
> mailbox API. Also convert the only user of PL320, highbank-cpufreq.c
> to work with thee API. Drop the obsoleted driver pl320-ipc.c
I think the conversion is fine based on your API, but you have
eliminated the stand-alone Rx interrupt code in the conversion. I
searched for if anybody is registering these rx atomic notifiers in 3.9,
and didn't find any. Is this expected to stay like this or is it some
future functionality not yet added, but getting removed in this patch.
>
> Signed-off-by: Jassi Brar <jaswinder.singh@linaro.org>
> ---
> drivers/cpufreq/highbank-cpufreq.c | 22 +++-
> drivers/mailbox/Makefile | 2 +-
> drivers/mailbox/{pl320-ipc.c => pl320.c} | 194 ++++++++++++++++--------------
> include/linux/pl320-ipc.h | 17 ---
> 4 files changed, 125 insertions(+), 110 deletions(-)
> rename drivers/mailbox/{pl320-ipc.c => pl320.c} (51%)
> delete mode 100644 include/linux/pl320-ipc.h
>
> diff --git a/drivers/cpufreq/highbank-cpufreq.c b/drivers/cpufreq/highbank-cpufreq.c
> index 3118b87..5c057e0 100644
> --- a/drivers/cpufreq/highbank-cpufreq.c
> +++ b/drivers/cpufreq/highbank-cpufreq.c
> @@ -19,7 +19,7 @@
> #include <linux/cpu.h>
> #include <linux/err.h>
> #include <linux/of.h>
> -#include <linux/pl320-ipc.h>
> +#include <linux/mailbox_client.h>
> #include <linux/platform_device.h>
>
> #define HB_CPUFREQ_CHANGE_NOTE 0x80000001
> @@ -29,8 +29,26 @@
> static int hb_voltage_change(unsigned int freq)
> {
> u32 msg[HB_CPUFREQ_IPC_LEN] = {HB_CPUFREQ_CHANGE_NOTE, freq / 1000000};
> + struct ipc_client cl;
> + int ret = -ETIMEDOUT;
> + void *chan;
>
> - return pl320_ipc_transmit(msg);
> + cl.rxcb = NULL;
> + cl.txcb = NULL;
> + cl.tx_block = true;
> + cl.tx_tout = 1000; /* 1 sec */
> + cl.cntlr_data = NULL;
> + cl.knows_txdone = false;
> + cl.chan_name = "pl320:A9_to_M3";
> +
> + chan = ipc_request_channel(&cl);
> +
> + if (ipc_send_message(chan, (void *)msg))
> + ret = msg[1]; /* PL320 updates buffer with FIFO after ACK */
> +
> + ipc_free_channel(chan);
I think I understand why you have done this, but do you really want to
request and free every time in the highbank cpufreq driver?
regards
Suman
^ permalink raw reply [flat|nested] 90+ messages in thread* Re: [RFC 3/3] mailbox: pl320: Introduce common API driver
@ 2013-04-29 16:44 ` Suman Anna
0 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-04-29 16:44 UTC (permalink / raw)
To: jassisinghbrar
Cc: loic.pallardy, arnd, linux-kernel, linux-arm-kernel, andy.green,
Jassi Brar, Mark Langsdorf
On 04/27/2013 01:14 PM, jassisinghbrar@gmail.com wrote:
> From: Jassi Brar <jaswinder.singh@linaro.org>
>
> Convert the PL320 controller driver to work with the common
> mailbox API. Also convert the only user of PL320, highbank-cpufreq.c
> to work with thee API. Drop the obsoleted driver pl320-ipc.c
I think the conversion is fine based on your API, but you have
eliminated the stand-alone Rx interrupt code in the conversion. I
searched for if anybody is registering these rx atomic notifiers in 3.9,
and didn't find any. Is this expected to stay like this or is it some
future functionality not yet added, but getting removed in this patch.
>
> Signed-off-by: Jassi Brar <jaswinder.singh@linaro.org>
> ---
> drivers/cpufreq/highbank-cpufreq.c | 22 +++-
> drivers/mailbox/Makefile | 2 +-
> drivers/mailbox/{pl320-ipc.c => pl320.c} | 194 ++++++++++++++++--------------
> include/linux/pl320-ipc.h | 17 ---
> 4 files changed, 125 insertions(+), 110 deletions(-)
> rename drivers/mailbox/{pl320-ipc.c => pl320.c} (51%)
> delete mode 100644 include/linux/pl320-ipc.h
>
> diff --git a/drivers/cpufreq/highbank-cpufreq.c b/drivers/cpufreq/highbank-cpufreq.c
> index 3118b87..5c057e0 100644
> --- a/drivers/cpufreq/highbank-cpufreq.c
> +++ b/drivers/cpufreq/highbank-cpufreq.c
> @@ -19,7 +19,7 @@
> #include <linux/cpu.h>
> #include <linux/err.h>
> #include <linux/of.h>
> -#include <linux/pl320-ipc.h>
> +#include <linux/mailbox_client.h>
> #include <linux/platform_device.h>
>
> #define HB_CPUFREQ_CHANGE_NOTE 0x80000001
> @@ -29,8 +29,26 @@
> static int hb_voltage_change(unsigned int freq)
> {
> u32 msg[HB_CPUFREQ_IPC_LEN] = {HB_CPUFREQ_CHANGE_NOTE, freq / 1000000};
> + struct ipc_client cl;
> + int ret = -ETIMEDOUT;
> + void *chan;
>
> - return pl320_ipc_transmit(msg);
> + cl.rxcb = NULL;
> + cl.txcb = NULL;
> + cl.tx_block = true;
> + cl.tx_tout = 1000; /* 1 sec */
> + cl.cntlr_data = NULL;
> + cl.knows_txdone = false;
> + cl.chan_name = "pl320:A9_to_M3";
> +
> + chan = ipc_request_channel(&cl);
> +
> + if (ipc_send_message(chan, (void *)msg))
> + ret = msg[1]; /* PL320 updates buffer with FIFO after ACK */
> +
> + ipc_free_channel(chan);
I think I understand why you have done this, but do you really want to
request and free every time in the highbank cpufreq driver?
regards
Suman
^ permalink raw reply [flat|nested] 90+ messages in thread* [RFC 3/3] mailbox: pl320: Introduce common API driver
2013-04-29 16:44 ` Suman Anna
@ 2013-04-29 16:57 ` Jassi Brar
-1 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-04-29 16:57 UTC (permalink / raw)
To: linux-arm-kernel
On 29 April 2013 22:14, Suman Anna <s-anna@ti.com> wrote:
> On 04/27/2013 01:14 PM, jassisinghbrar at gmail.com wrote:
>> From: Jassi Brar <jaswinder.singh@linaro.org>
>>
>> Convert the PL320 controller driver to work with the common
>> mailbox API. Also convert the only user of PL320, highbank-cpufreq.c
>> to work with thee API. Drop the obsoleted driver pl320-ipc.c
>
> I think the conversion is fine based on your API, but you have
> eliminated the stand-alone Rx interrupt code in the conversion. I
> searched for if anybody is registering these rx atomic notifiers in 3.9,
> and didn't find any. Is this expected to stay like this or is it some
> future functionality not yet added, but getting removed in this patch.
>
Yeah, probably only some out-of-tree code needed that but that made my
life simpler :)
>>
>> Signed-off-by: Jassi Brar <jaswinder.singh@linaro.org>
>> ---
>> drivers/cpufreq/highbank-cpufreq.c | 22 +++-
>> drivers/mailbox/Makefile | 2 +-
>> drivers/mailbox/{pl320-ipc.c => pl320.c} | 194 ++++++++++++++++--------------
>> include/linux/pl320-ipc.h | 17 ---
>> 4 files changed, 125 insertions(+), 110 deletions(-)
>> rename drivers/mailbox/{pl320-ipc.c => pl320.c} (51%)
>> delete mode 100644 include/linux/pl320-ipc.h
>>
>> diff --git a/drivers/cpufreq/highbank-cpufreq.c b/drivers/cpufreq/highbank-cpufreq.c
>> index 3118b87..5c057e0 100644
>> --- a/drivers/cpufreq/highbank-cpufreq.c
>> +++ b/drivers/cpufreq/highbank-cpufreq.c
>> @@ -19,7 +19,7 @@
>> #include <linux/cpu.h>
>> #include <linux/err.h>
>> #include <linux/of.h>
>> -#include <linux/pl320-ipc.h>
>> +#include <linux/mailbox_client.h>
>> #include <linux/platform_device.h>
>>
>> #define HB_CPUFREQ_CHANGE_NOTE 0x80000001
>> @@ -29,8 +29,26 @@
>> static int hb_voltage_change(unsigned int freq)
>> {
>> u32 msg[HB_CPUFREQ_IPC_LEN] = {HB_CPUFREQ_CHANGE_NOTE, freq / 1000000};
>> + struct ipc_client cl;
>> + int ret = -ETIMEDOUT;
>> + void *chan;
>>
>> - return pl320_ipc_transmit(msg);
>> + cl.rxcb = NULL;
>> + cl.txcb = NULL;
>> + cl.tx_block = true;
>> + cl.tx_tout = 1000; /* 1 sec */
>> + cl.cntlr_data = NULL;
>> + cl.knows_txdone = false;
>> + cl.chan_name = "pl320:A9_to_M3";
>> +
>> + chan = ipc_request_channel(&cl);
>> +
>> + if (ipc_send_message(chan, (void *)msg))
>> + ret = msg[1]; /* PL320 updates buffer with FIFO after ACK */
>> +
>> + ipc_free_channel(chan);
>
> I think I understand why you have done this, but do you really want to
> request and free every time in the highbank cpufreq driver?
>
Exactly my aim - make the API light enough to enable the client to
use-and-throw. And also because the channel are exclusively assigned,
acquire a channel only for as long as you need it.
cheers.
^ permalink raw reply [flat|nested] 90+ messages in thread* Re: [RFC 3/3] mailbox: pl320: Introduce common API driver
@ 2013-04-29 16:57 ` Jassi Brar
0 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-04-29 16:57 UTC (permalink / raw)
To: Suman Anna
Cc: Jassi Brar, Loic PALLARDY (loic.pallardy@st.com), Arnd Bergmann,
lkml, linux-arm-kernel@lists.infradead.org, Andy Green,
Mark Langsdorf
On 29 April 2013 22:14, Suman Anna <s-anna@ti.com> wrote:
> On 04/27/2013 01:14 PM, jassisinghbrar@gmail.com wrote:
>> From: Jassi Brar <jaswinder.singh@linaro.org>
>>
>> Convert the PL320 controller driver to work with the common
>> mailbox API. Also convert the only user of PL320, highbank-cpufreq.c
>> to work with thee API. Drop the obsoleted driver pl320-ipc.c
>
> I think the conversion is fine based on your API, but you have
> eliminated the stand-alone Rx interrupt code in the conversion. I
> searched for if anybody is registering these rx atomic notifiers in 3.9,
> and didn't find any. Is this expected to stay like this or is it some
> future functionality not yet added, but getting removed in this patch.
>
Yeah, probably only some out-of-tree code needed that but that made my
life simpler :)
>>
>> Signed-off-by: Jassi Brar <jaswinder.singh@linaro.org>
>> ---
>> drivers/cpufreq/highbank-cpufreq.c | 22 +++-
>> drivers/mailbox/Makefile | 2 +-
>> drivers/mailbox/{pl320-ipc.c => pl320.c} | 194 ++++++++++++++++--------------
>> include/linux/pl320-ipc.h | 17 ---
>> 4 files changed, 125 insertions(+), 110 deletions(-)
>> rename drivers/mailbox/{pl320-ipc.c => pl320.c} (51%)
>> delete mode 100644 include/linux/pl320-ipc.h
>>
>> diff --git a/drivers/cpufreq/highbank-cpufreq.c b/drivers/cpufreq/highbank-cpufreq.c
>> index 3118b87..5c057e0 100644
>> --- a/drivers/cpufreq/highbank-cpufreq.c
>> +++ b/drivers/cpufreq/highbank-cpufreq.c
>> @@ -19,7 +19,7 @@
>> #include <linux/cpu.h>
>> #include <linux/err.h>
>> #include <linux/of.h>
>> -#include <linux/pl320-ipc.h>
>> +#include <linux/mailbox_client.h>
>> #include <linux/platform_device.h>
>>
>> #define HB_CPUFREQ_CHANGE_NOTE 0x80000001
>> @@ -29,8 +29,26 @@
>> static int hb_voltage_change(unsigned int freq)
>> {
>> u32 msg[HB_CPUFREQ_IPC_LEN] = {HB_CPUFREQ_CHANGE_NOTE, freq / 1000000};
>> + struct ipc_client cl;
>> + int ret = -ETIMEDOUT;
>> + void *chan;
>>
>> - return pl320_ipc_transmit(msg);
>> + cl.rxcb = NULL;
>> + cl.txcb = NULL;
>> + cl.tx_block = true;
>> + cl.tx_tout = 1000; /* 1 sec */
>> + cl.cntlr_data = NULL;
>> + cl.knows_txdone = false;
>> + cl.chan_name = "pl320:A9_to_M3";
>> +
>> + chan = ipc_request_channel(&cl);
>> +
>> + if (ipc_send_message(chan, (void *)msg))
>> + ret = msg[1]; /* PL320 updates buffer with FIFO after ACK */
>> +
>> + ipc_free_channel(chan);
>
> I think I understand why you have done this, but do you really want to
> request and free every time in the highbank cpufreq driver?
>
Exactly my aim - make the API light enough to enable the client to
use-and-throw. And also because the channel are exclusively assigned,
acquire a channel only for as long as you need it.
cheers.
^ permalink raw reply [flat|nested] 90+ messages in thread* [RFC 3/3] mailbox: pl320: Introduce common API driver
2013-04-29 16:57 ` Jassi Brar
@ 2013-04-29 17:06 ` Mark Langsdorf
-1 siblings, 0 replies; 90+ messages in thread
From: Mark Langsdorf @ 2013-04-29 17:06 UTC (permalink / raw)
To: linux-arm-kernel
On 04/29/2013 11:57 AM, Jassi Brar wrote:
> On 29 April 2013 22:14, Suman Anna <s-anna@ti.com> wrote:
>> On 04/27/2013 01:14 PM, jassisinghbrar at gmail.com wrote:
>>> From: Jassi Brar <jaswinder.singh@linaro.org>
>>> Signed-off-by: Jassi Brar <jaswinder.singh@linaro.org>
>>> ---
>>> drivers/cpufreq/highbank-cpufreq.c | 22 +++-
>>> drivers/mailbox/Makefile | 2 +-
>>> drivers/mailbox/{pl320-ipc.c => pl320.c} | 194 ++++++++++++++++--------------
>>> include/linux/pl320-ipc.h | 17 ---
>>> 4 files changed, 125 insertions(+), 110 deletions(-)
>>> rename drivers/mailbox/{pl320-ipc.c => pl320.c} (51%)
>>> delete mode 100644 include/linux/pl320-ipc.h
>>>
>>> diff --git a/drivers/cpufreq/highbank-cpufreq.c b/drivers/cpufreq/highbank-cpufreq.c
>>> index 3118b87..5c057e0 100644
>>> --- a/drivers/cpufreq/highbank-cpufreq.c
>>> +++ b/drivers/cpufreq/highbank-cpufreq.c
>>> @@ -19,7 +19,7 @@
>>> #include <linux/cpu.h>
>>> #include <linux/err.h>
>>> #include <linux/of.h>
>>> -#include <linux/pl320-ipc.h>
>>> +#include <linux/mailbox_client.h>
>>> #include <linux/platform_device.h>
>>>
>>> #define HB_CPUFREQ_CHANGE_NOTE 0x80000001
>>> @@ -29,8 +29,26 @@
>>> static int hb_voltage_change(unsigned int freq)
>>> {
>>> u32 msg[HB_CPUFREQ_IPC_LEN] = {HB_CPUFREQ_CHANGE_NOTE, freq / 1000000};
>>> + struct ipc_client cl;
>>> + int ret = -ETIMEDOUT;
>>> + void *chan;
>>>
>>> - return pl320_ipc_transmit(msg);
>>> + cl.rxcb = NULL;
>>> + cl.txcb = NULL;
>>> + cl.tx_block = true;
>>> + cl.tx_tout = 1000; /* 1 sec */
>>> + cl.cntlr_data = NULL;
>>> + cl.knows_txdone = false;
>>> + cl.chan_name = "pl320:A9_to_M3";
>>> +
>>> + chan = ipc_request_channel(&cl);
>>> +
>>> + if (ipc_send_message(chan, (void *)msg))
>>> + ret = msg[1]; /* PL320 updates buffer with FIFO after ACK */
>>> +
>>> + ipc_free_channel(chan);
>>
>> I think I understand why you have done this, but do you really want to
>> request and free every time in the highbank cpufreq driver?
>>
> Exactly my aim - make the API light enough to enable the client to
> use-and-throw. And also because the channel are exclusively assigned,
> acquire a channel only for as long as you need it.
Do you have any numbers on the performance impact? Our cpufreq
transition throughput is bad enough without adding additional delays.
--Mark Langsdorf
Calxeda, Inc.
^ permalink raw reply [flat|nested] 90+ messages in thread* Re: [RFC 3/3] mailbox: pl320: Introduce common API driver
@ 2013-04-29 17:06 ` Mark Langsdorf
0 siblings, 0 replies; 90+ messages in thread
From: Mark Langsdorf @ 2013-04-29 17:06 UTC (permalink / raw)
To: Jassi Brar
Cc: Suman Anna, Jassi Brar, Loic PALLARDY (loic.pallardy@st.com),
Arnd Bergmann, lkml, linux-arm-kernel@lists.infradead.org,
Andy Green
On 04/29/2013 11:57 AM, Jassi Brar wrote:
> On 29 April 2013 22:14, Suman Anna <s-anna@ti.com> wrote:
>> On 04/27/2013 01:14 PM, jassisinghbrar@gmail.com wrote:
>>> From: Jassi Brar <jaswinder.singh@linaro.org>
>>> Signed-off-by: Jassi Brar <jaswinder.singh@linaro.org>
>>> ---
>>> drivers/cpufreq/highbank-cpufreq.c | 22 +++-
>>> drivers/mailbox/Makefile | 2 +-
>>> drivers/mailbox/{pl320-ipc.c => pl320.c} | 194 ++++++++++++++++--------------
>>> include/linux/pl320-ipc.h | 17 ---
>>> 4 files changed, 125 insertions(+), 110 deletions(-)
>>> rename drivers/mailbox/{pl320-ipc.c => pl320.c} (51%)
>>> delete mode 100644 include/linux/pl320-ipc.h
>>>
>>> diff --git a/drivers/cpufreq/highbank-cpufreq.c b/drivers/cpufreq/highbank-cpufreq.c
>>> index 3118b87..5c057e0 100644
>>> --- a/drivers/cpufreq/highbank-cpufreq.c
>>> +++ b/drivers/cpufreq/highbank-cpufreq.c
>>> @@ -19,7 +19,7 @@
>>> #include <linux/cpu.h>
>>> #include <linux/err.h>
>>> #include <linux/of.h>
>>> -#include <linux/pl320-ipc.h>
>>> +#include <linux/mailbox_client.h>
>>> #include <linux/platform_device.h>
>>>
>>> #define HB_CPUFREQ_CHANGE_NOTE 0x80000001
>>> @@ -29,8 +29,26 @@
>>> static int hb_voltage_change(unsigned int freq)
>>> {
>>> u32 msg[HB_CPUFREQ_IPC_LEN] = {HB_CPUFREQ_CHANGE_NOTE, freq / 1000000};
>>> + struct ipc_client cl;
>>> + int ret = -ETIMEDOUT;
>>> + void *chan;
>>>
>>> - return pl320_ipc_transmit(msg);
>>> + cl.rxcb = NULL;
>>> + cl.txcb = NULL;
>>> + cl.tx_block = true;
>>> + cl.tx_tout = 1000; /* 1 sec */
>>> + cl.cntlr_data = NULL;
>>> + cl.knows_txdone = false;
>>> + cl.chan_name = "pl320:A9_to_M3";
>>> +
>>> + chan = ipc_request_channel(&cl);
>>> +
>>> + if (ipc_send_message(chan, (void *)msg))
>>> + ret = msg[1]; /* PL320 updates buffer with FIFO after ACK */
>>> +
>>> + ipc_free_channel(chan);
>>
>> I think I understand why you have done this, but do you really want to
>> request and free every time in the highbank cpufreq driver?
>>
> Exactly my aim - make the API light enough to enable the client to
> use-and-throw. And also because the channel are exclusively assigned,
> acquire a channel only for as long as you need it.
Do you have any numbers on the performance impact? Our cpufreq
transition throughput is bad enough without adding additional delays.
--Mark Langsdorf
Calxeda, Inc.
^ permalink raw reply [flat|nested] 90+ messages in thread* [RFC 3/3] mailbox: pl320: Introduce common API driver
2013-04-29 17:06 ` Mark Langsdorf
@ 2013-04-29 17:28 ` Jassi Brar
-1 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-04-29 17:28 UTC (permalink / raw)
To: linux-arm-kernel
On 29 April 2013 22:36, Mark Langsdorf <mark.langsdorf@calxeda.com> wrote:
> On 04/29/2013 11:57 AM, Jassi Brar wrote:
>> On 29 April 2013 22:14, Suman Anna <s-anna@ti.com> wrote:
>>> On 04/27/2013 01:14 PM, jassisinghbrar at gmail.com wrote:
>>>> From: Jassi Brar <jaswinder.singh@linaro.org>
>>>> Signed-off-by: Jassi Brar <jaswinder.singh@linaro.org>
>>>> ---
>>>> drivers/cpufreq/highbank-cpufreq.c | 22 +++-
>>>> drivers/mailbox/Makefile | 2 +-
>>>> drivers/mailbox/{pl320-ipc.c => pl320.c} | 194 ++++++++++++++++--------------
>>>> include/linux/pl320-ipc.h | 17 ---
>>>> 4 files changed, 125 insertions(+), 110 deletions(-)
>>>> rename drivers/mailbox/{pl320-ipc.c => pl320.c} (51%)
>>>> delete mode 100644 include/linux/pl320-ipc.h
>>>>
>>>> diff --git a/drivers/cpufreq/highbank-cpufreq.c b/drivers/cpufreq/highbank-cpufreq.c
>>>> index 3118b87..5c057e0 100644
>>>> --- a/drivers/cpufreq/highbank-cpufreq.c
>>>> +++ b/drivers/cpufreq/highbank-cpufreq.c
>>>> @@ -19,7 +19,7 @@
>>>> #include <linux/cpu.h>
>>>> #include <linux/err.h>
>>>> #include <linux/of.h>
>>>> -#include <linux/pl320-ipc.h>
>>>> +#include <linux/mailbox_client.h>
>>>> #include <linux/platform_device.h>
>>>>
>>>> #define HB_CPUFREQ_CHANGE_NOTE 0x80000001
>>>> @@ -29,8 +29,26 @@
>>>> static int hb_voltage_change(unsigned int freq)
>>>> {
>>>> u32 msg[HB_CPUFREQ_IPC_LEN] = {HB_CPUFREQ_CHANGE_NOTE, freq / 1000000};
>>>> + struct ipc_client cl;
>>>> + int ret = -ETIMEDOUT;
>>>> + void *chan;
>>>>
>>>> - return pl320_ipc_transmit(msg);
>>>> + cl.rxcb = NULL;
>>>> + cl.txcb = NULL;
>>>> + cl.tx_block = true;
>>>> + cl.tx_tout = 1000; /* 1 sec */
>>>> + cl.cntlr_data = NULL;
>>>> + cl.knows_txdone = false;
>>>> + cl.chan_name = "pl320:A9_to_M3";
>>>> +
>>>> + chan = ipc_request_channel(&cl);
>>>> +
>>>> + if (ipc_send_message(chan, (void *)msg))
>>>> + ret = msg[1]; /* PL320 updates buffer with FIFO after ACK */
>>>> +
>>>> + ipc_free_channel(chan);
>>>
>>> I think I understand why you have done this, but do you really want to
>>> request and free every time in the highbank cpufreq driver?
>>>
>> Exactly my aim - make the API light enough to enable the client to
>> use-and-throw. And also because the channel are exclusively assigned,
>> acquire a channel only for as long as you need it.
>
> Do you have any numbers on the performance impact? Our cpufreq
> transition throughput is bad enough without adding additional delays.
>
Sorry no numbers.
However if you look closely you'll find for your usecase the message
directly reaches the controller and the reply from remote is directly
handed to your client driver. And since there is no context for
channel on your platform channel request/free shouldn't fail either
(or you could keep the channel allocated for the lifetime of the
client driver). So I would not expect any longer delays than the
original way.
Compared with TI's framework of RX-via-notifier you should be better
off with this API, imho.
cheers.
^ permalink raw reply [flat|nested] 90+ messages in thread* Re: [RFC 3/3] mailbox: pl320: Introduce common API driver
@ 2013-04-29 17:28 ` Jassi Brar
0 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-04-29 17:28 UTC (permalink / raw)
To: Mark Langsdorf
Cc: Suman Anna, Jassi Brar, Loic PALLARDY (loic.pallardy@st.com),
Arnd Bergmann, lkml, linux-arm-kernel@lists.infradead.org,
Andy Green
On 29 April 2013 22:36, Mark Langsdorf <mark.langsdorf@calxeda.com> wrote:
> On 04/29/2013 11:57 AM, Jassi Brar wrote:
>> On 29 April 2013 22:14, Suman Anna <s-anna@ti.com> wrote:
>>> On 04/27/2013 01:14 PM, jassisinghbrar@gmail.com wrote:
>>>> From: Jassi Brar <jaswinder.singh@linaro.org>
>>>> Signed-off-by: Jassi Brar <jaswinder.singh@linaro.org>
>>>> ---
>>>> drivers/cpufreq/highbank-cpufreq.c | 22 +++-
>>>> drivers/mailbox/Makefile | 2 +-
>>>> drivers/mailbox/{pl320-ipc.c => pl320.c} | 194 ++++++++++++++++--------------
>>>> include/linux/pl320-ipc.h | 17 ---
>>>> 4 files changed, 125 insertions(+), 110 deletions(-)
>>>> rename drivers/mailbox/{pl320-ipc.c => pl320.c} (51%)
>>>> delete mode 100644 include/linux/pl320-ipc.h
>>>>
>>>> diff --git a/drivers/cpufreq/highbank-cpufreq.c b/drivers/cpufreq/highbank-cpufreq.c
>>>> index 3118b87..5c057e0 100644
>>>> --- a/drivers/cpufreq/highbank-cpufreq.c
>>>> +++ b/drivers/cpufreq/highbank-cpufreq.c
>>>> @@ -19,7 +19,7 @@
>>>> #include <linux/cpu.h>
>>>> #include <linux/err.h>
>>>> #include <linux/of.h>
>>>> -#include <linux/pl320-ipc.h>
>>>> +#include <linux/mailbox_client.h>
>>>> #include <linux/platform_device.h>
>>>>
>>>> #define HB_CPUFREQ_CHANGE_NOTE 0x80000001
>>>> @@ -29,8 +29,26 @@
>>>> static int hb_voltage_change(unsigned int freq)
>>>> {
>>>> u32 msg[HB_CPUFREQ_IPC_LEN] = {HB_CPUFREQ_CHANGE_NOTE, freq / 1000000};
>>>> + struct ipc_client cl;
>>>> + int ret = -ETIMEDOUT;
>>>> + void *chan;
>>>>
>>>> - return pl320_ipc_transmit(msg);
>>>> + cl.rxcb = NULL;
>>>> + cl.txcb = NULL;
>>>> + cl.tx_block = true;
>>>> + cl.tx_tout = 1000; /* 1 sec */
>>>> + cl.cntlr_data = NULL;
>>>> + cl.knows_txdone = false;
>>>> + cl.chan_name = "pl320:A9_to_M3";
>>>> +
>>>> + chan = ipc_request_channel(&cl);
>>>> +
>>>> + if (ipc_send_message(chan, (void *)msg))
>>>> + ret = msg[1]; /* PL320 updates buffer with FIFO after ACK */
>>>> +
>>>> + ipc_free_channel(chan);
>>>
>>> I think I understand why you have done this, but do you really want to
>>> request and free every time in the highbank cpufreq driver?
>>>
>> Exactly my aim - make the API light enough to enable the client to
>> use-and-throw. And also because the channel are exclusively assigned,
>> acquire a channel only for as long as you need it.
>
> Do you have any numbers on the performance impact? Our cpufreq
> transition throughput is bad enough without adding additional delays.
>
Sorry no numbers.
However if you look closely you'll find for your usecase the message
directly reaches the controller and the reply from remote is directly
handed to your client driver. And since there is no context for
channel on your platform channel request/free shouldn't fail either
(or you could keep the channel allocated for the lifetime of the
client driver). So I would not expect any longer delays than the
original way.
Compared with TI's framework of RX-via-notifier you should be better
off with this API, imho.
cheers.
^ permalink raw reply [flat|nested] 90+ messages in thread
* [PATCHv3 00/14] drivers: mailbox: framework creation
2013-04-27 4:51 ` Jassi Brar
@ 2013-04-29 16:00 ` Suman Anna
-1 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-04-29 16:00 UTC (permalink / raw)
To: linux-arm-kernel
Hi Jassi,
On 04/26/2013 11:51 PM, Jassi Brar wrote:
> Hi Suman,
>
>>> On 26 April 2013 03:59, Suman Anna <s-anna@ti.com> wrote:
>>>> On 04/25/2013 12:20 AM, Jassi Brar wrote:
>
>>> I never said no-buffering and I never said buffering should be in
>>> controller drivers. In fact I don't remember ever objecting to how
>>> buffering is done in TI's framework.
>>> A controller could service only 1 request at a time so lets give it
>>> just 1 at a time. Let the API handle the complexity of buffering.
>>>
>>
>> Alright, guess this got lost in translation :). I interpreted based on
>> the fact that you wanted to get rid of the size field from the
>> mailbox_msg definition. Do you have a different mechanism in mind for
>> the buffering compared to the present one?
>>
> Sure, a very simple but efficient one. I had started on pseudo code
> implementation the day I first replied, but now I have real code with
> the PL320 controller and the Highbank client converted to the API. All
> that I say features in the new design. Polishing and documentation
> will take just a few hours more. You could see end to end what I have
> been talking about.
>>
>> OK, I didn't think of a no RTR interrupt-based controller. I would thing
>> that such a controller is very rudimentary. I wonder if there are any
>> controllers like this out there.
>>
> One of my controllers is like that :)
I hope it does have a status register atleast, and not the "neither
report nor sense RTR" type.
>
>>>
>>> BTW, TI's RX mechanism too seems broken for common API. Receiving
>>> every few bytes via 'notify' mechanism is very inefficient. Imagine a
>>> platform with no shared memory between co-processors and the local
>>> wants to diagnose the remote by asking critical data at least KBs in
>>> size.
>>
>> No shared memory between co-processors and a relatively slow wire
>> transport is a bad architecture design to begin with.
>>
> IMHO it's only about private memory. Even if the controller transfers,
> say, 10bytes/interrupt there could always be a requirement to read
> some 1MB region of remote's private memory. And the same logic implies
> that our TX too should be as fast as possible - the remote might need
> its 1MB firmware over the link. So let us just try to serve all
> designs rather than evaluate them :)
>
>
>>> So when API has nothing to do with received packet and the controller
>>> has to get rid of it asap so as to be able to receive the next, IMHO
>>> there should be short-circuit from controller to client via the API.
>>> No delay, no buffering of RX.
>>
>> The current TI design is based on the fact that we can get multiple
>> messages on a single interrupt due to the h/w fifo and the driver takes
>> care of the bottom-half. Leaving it to the client is putting a lot of
>> faith in the client and doesn't scale to multiple clients. The client
>> would have to perform mostly the same as the driver is doing - so this
>> goes back to the base discussion point that we have - which is the lack
>> of support for atomic_context receivers in the current code. I perceive
>> this as an attribute of the controller/mailbox device itself rather than
>> the client.
>>
> Sorry, I don't understand the concern about faith.
> If the controller h/w absolutely can not tell the remote(sender) of a
> received packet (as seems to be your case), its driver shouldn't even
> try to demux the received messages. The client driver must know which
> remotes could send it a message and how to discern them on the
> platform. Some 'server' RX client is needed here.
No demuxing, deliver the message to the different clients. It is a
protocol agreement between the clients on what the message means. Think
of this scenario akin to shared interrupts.
> If the controller could indeed map received packet onto remotes, then
> ideally the controller driver should declare one (RX only) channel for
> each such remote and demux packets onto them.
> In either case, 'notify' mechanism is not necessary.
The notify mechanism was the top-half on the interrupt handling. The
faith part is coming from the fact that you expect all the clients to do
the equivalent of the bottom-half (which would mean some duplication in
the different clients), the OMAP scenario is such that all the different
link interrupts (both rx & tx) are mapped onto a single physical
interrupt. I think this may not be applicable to your usecase, wherein
you probably expect a response back before proceeding.
>
>
>> I agree that all remote-ends will not
>> be able to cope up intermixed requests, but isn't this again a
>> controller architecture dependent?
>>
> I think it's more about remote's protocol implementation than
> controller's architecture.
Right, I meant functional integration.
> If tomorrow TI's remote firmware introduces a new set of critical
> commands that may arrive only in a particular sequence, you'll find
> yourself sharing a ride on our dinghy :)
> And Andy already explained where we come from.
This is almost always true when your remote is for offloading some h/w
operations.
regards
Suman
^ permalink raw reply [flat|nested] 90+ messages in thread
* Re: [PATCHv3 00/14] drivers: mailbox: framework creation
@ 2013-04-29 16:00 ` Suman Anna
0 siblings, 0 replies; 90+ messages in thread
From: Suman Anna @ 2013-04-29 16:00 UTC (permalink / raw)
To: Jassi Brar
Cc: Loic PALLARDY, Jassi Brar, Ohad Ben-Cohen (ohad@wizery.com),
Stephen Rothwell, Andy Green (andy.green@linaro.org),
Russell King, Arnd Bergmann, Tony Lindgren, Greg Kroah-Hartman,
Linus Walleij, Rafael J. Wysocki, Linux Kernel Mailing List,
Omar Ramirez Luna (omar.ramirez@copitl.com),
linux-arm-kernel@lists.infradead.org
Hi Jassi,
On 04/26/2013 11:51 PM, Jassi Brar wrote:
> Hi Suman,
>
>>> On 26 April 2013 03:59, Suman Anna <s-anna@ti.com> wrote:
>>>> On 04/25/2013 12:20 AM, Jassi Brar wrote:
>
>>> I never said no-buffering and I never said buffering should be in
>>> controller drivers. In fact I don't remember ever objecting to how
>>> buffering is done in TI's framework.
>>> A controller could service only 1 request at a time so lets give it
>>> just 1 at a time. Let the API handle the complexity of buffering.
>>>
>>
>> Alright, guess this got lost in translation :). I interpreted based on
>> the fact that you wanted to get rid of the size field from the
>> mailbox_msg definition. Do you have a different mechanism in mind for
>> the buffering compared to the present one?
>>
> Sure, a very simple but efficient one. I had started on pseudo code
> implementation the day I first replied, but now I have real code with
> the PL320 controller and the Highbank client converted to the API. All
> that I say features in the new design. Polishing and documentation
> will take just a few hours more. You could see end to end what I have
> been talking about.
>>
>> OK, I didn't think of a no RTR interrupt-based controller. I would thing
>> that such a controller is very rudimentary. I wonder if there are any
>> controllers like this out there.
>>
> One of my controllers is like that :)
I hope it does have a status register atleast, and not the "neither
report nor sense RTR" type.
>
>>>
>>> BTW, TI's RX mechanism too seems broken for common API. Receiving
>>> every few bytes via 'notify' mechanism is very inefficient. Imagine a
>>> platform with no shared memory between co-processors and the local
>>> wants to diagnose the remote by asking critical data at least KBs in
>>> size.
>>
>> No shared memory between co-processors and a relatively slow wire
>> transport is a bad architecture design to begin with.
>>
> IMHO it's only about private memory. Even if the controller transfers,
> say, 10bytes/interrupt there could always be a requirement to read
> some 1MB region of remote's private memory. And the same logic implies
> that our TX too should be as fast as possible - the remote might need
> its 1MB firmware over the link. So let us just try to serve all
> designs rather than evaluate them :)
>
>
>>> So when API has nothing to do with received packet and the controller
>>> has to get rid of it asap so as to be able to receive the next, IMHO
>>> there should be short-circuit from controller to client via the API.
>>> No delay, no buffering of RX.
>>
>> The current TI design is based on the fact that we can get multiple
>> messages on a single interrupt due to the h/w fifo and the driver takes
>> care of the bottom-half. Leaving it to the client is putting a lot of
>> faith in the client and doesn't scale to multiple clients. The client
>> would have to perform mostly the same as the driver is doing - so this
>> goes back to the base discussion point that we have - which is the lack
>> of support for atomic_context receivers in the current code. I perceive
>> this as an attribute of the controller/mailbox device itself rather than
>> the client.
>>
> Sorry, I don't understand the concern about faith.
> If the controller h/w absolutely can not tell the remote(sender) of a
> received packet (as seems to be your case), its driver shouldn't even
> try to demux the received messages. The client driver must know which
> remotes could send it a message and how to discern them on the
> platform. Some 'server' RX client is needed here.
No demuxing, deliver the message to the different clients. It is a
protocol agreement between the clients on what the message means. Think
of this scenario akin to shared interrupts.
> If the controller could indeed map received packet onto remotes, then
> ideally the controller driver should declare one (RX only) channel for
> each such remote and demux packets onto them.
> In either case, 'notify' mechanism is not necessary.
The notify mechanism was the top-half on the interrupt handling. The
faith part is coming from the fact that you expect all the clients to do
the equivalent of the bottom-half (which would mean some duplication in
the different clients), the OMAP scenario is such that all the different
link interrupts (both rx & tx) are mapped onto a single physical
interrupt. I think this may not be applicable to your usecase, wherein
you probably expect a response back before proceeding.
>
>
>> I agree that all remote-ends will not
>> be able to cope up intermixed requests, but isn't this again a
>> controller architecture dependent?
>>
> I think it's more about remote's protocol implementation than
> controller's architecture.
Right, I meant functional integration.
> If tomorrow TI's remote firmware introduces a new set of critical
> commands that may arrive only in a particular sequence, you'll find
> yourself sharing a ride on our dinghy :)
> And Andy already explained where we come from.
This is almost always true when your remote is for offloading some h/w
operations.
regards
Suman
^ permalink raw reply [flat|nested] 90+ messages in thread
* [PATCHv3 00/14] drivers: mailbox: framework creation
2013-04-29 16:00 ` Suman Anna
@ 2013-04-29 16:49 ` Jassi Brar
-1 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-04-29 16:49 UTC (permalink / raw)
To: linux-arm-kernel
Hi
On 29 April 2013 21:30, Suman Anna <s-anna@ti.com> wrote:
> Hi Jassi,
>
> On 04/26/2013 11:51 PM, Jassi Brar wrote:
>>> OK, I didn't think of a no RTR interrupt-based controller. I would thing
>>> that such a controller is very rudimentary. I wonder if there are any
>>> controllers like this out there.
>>>
>> One of my controllers is like that :)
>
> I hope it does have a status register atleast, and not the "neither
> report nor sense RTR" type.
>
Actually the status is not set by the h/w, but the remote's firmware
implementation makes sure it sets a marker in the status register
after accepting data. So with some other firmware, we might not even
have the status read facility and the client will have to solely run
the TX ticker.
>> If the controller h/w absolutely can not tell the remote(sender) of a
>> received packet (as seems to be your case), its driver shouldn't even
>> try to demux the received messages. The client driver must know which
>> remotes could send it a message and how to discern them on the
>> platform. Some 'server' RX client is needed here.
>
> No demuxing, deliver the message to the different clients. It is a
> protocol agreement between the clients on what the message means. Think
> of this scenario akin to shared interrupts.
>
Please re-read. That's what I said - No demuxing by the controller driver :)
>
> The notify mechanism was the top-half on the interrupt handling. The
> faith part is coming from the fact that you expect all the clients to do
> the equivalent of the bottom-half (which would mean some duplication in
> the different clients), the OMAP scenario is such that all the different
> link interrupts (both rx & tx) are mapped onto a single physical
> interrupt. I think this may not be applicable to your usecase, wherein
> you probably expect a response back before proceeding.
>
Simply put - the RX by notifier method will fail should a client is
speed critical.
The api might provide it optionally, but direct handover should be the
default option.
Btw, I did put up an almost tested version of an API implementing most
of the features we agree upon
http://www.spinics.net/lists/kernel/msg1523873.html
http://www.spinics.net/lists/kernel/msg1523874.html
cheers.
^ permalink raw reply [flat|nested] 90+ messages in thread
* Re: [PATCHv3 00/14] drivers: mailbox: framework creation
@ 2013-04-29 16:49 ` Jassi Brar
0 siblings, 0 replies; 90+ messages in thread
From: Jassi Brar @ 2013-04-29 16:49 UTC (permalink / raw)
To: Suman Anna
Cc: Loic PALLARDY, Jassi Brar, Ohad Ben-Cohen (ohad@wizery.com),
Stephen Rothwell, Andy Green (andy.green@linaro.org),
Russell King, Arnd Bergmann, Tony Lindgren, Greg Kroah-Hartman,
Linus Walleij, Rafael J. Wysocki, Linux Kernel Mailing List,
Omar Ramirez Luna (omar.ramirez@copitl.com),
linux-arm-kernel@lists.infradead.org
Hi
On 29 April 2013 21:30, Suman Anna <s-anna@ti.com> wrote:
> Hi Jassi,
>
> On 04/26/2013 11:51 PM, Jassi Brar wrote:
>>> OK, I didn't think of a no RTR interrupt-based controller. I would thing
>>> that such a controller is very rudimentary. I wonder if there are any
>>> controllers like this out there.
>>>
>> One of my controllers is like that :)
>
> I hope it does have a status register atleast, and not the "neither
> report nor sense RTR" type.
>
Actually the status is not set by the h/w, but the remote's firmware
implementation makes sure it sets a marker in the status register
after accepting data. So with some other firmware, we might not even
have the status read facility and the client will have to solely run
the TX ticker.
>> If the controller h/w absolutely can not tell the remote(sender) of a
>> received packet (as seems to be your case), its driver shouldn't even
>> try to demux the received messages. The client driver must know which
>> remotes could send it a message and how to discern them on the
>> platform. Some 'server' RX client is needed here.
>
> No demuxing, deliver the message to the different clients. It is a
> protocol agreement between the clients on what the message means. Think
> of this scenario akin to shared interrupts.
>
Please re-read. That's what I said - No demuxing by the controller driver :)
>
> The notify mechanism was the top-half on the interrupt handling. The
> faith part is coming from the fact that you expect all the clients to do
> the equivalent of the bottom-half (which would mean some duplication in
> the different clients), the OMAP scenario is such that all the different
> link interrupts (both rx & tx) are mapped onto a single physical
> interrupt. I think this may not be applicable to your usecase, wherein
> you probably expect a response back before proceeding.
>
Simply put - the RX by notifier method will fail should a client is
speed critical.
The api might provide it optionally, but direct handover should be the
default option.
Btw, I did put up an almost tested version of an API implementing most
of the features we agree upon
http://www.spinics.net/lists/kernel/msg1523873.html
http://www.spinics.net/lists/kernel/msg1523874.html
cheers.
^ permalink raw reply [flat|nested] 90+ messages in thread