Re: [RFC][PATCH] KBUS messaging subsystem

Re: [RFC][PATCH] KBUS messaging subsystem

[Grant Likely]

On Fri, Jan 28, 2011 at 06:01:58PM +0000, Tony Ibbs wrote:
> We talked briefly at ELCE 2010 in Cambridge, after the "keeping
> kernel maintainers happy" talk, and you offered that I could
> CC you when I got round to submitting our kernel module.

Hi Tony,

Yes, I remember talking to you.  I'll take a look through and see if I
see any glaring problems.  However, my major critique still stands.
There are already a large number of communication channels available
to Linux, with DBUS gatting the most attention at the moment.  What
niche is KBUS filling that DBUS and others misses?  Secondarily, does
KBUS implement a new API instead of extending an existing mechanism to
meet your needs?  I suspect you'll get a lot more traction if KBUS can
make DBUS work better/faster/more efficiently.

> I've just emailed the embedded linux mailing list with an initial
> question about it, and attempted to CC you as
> grant.likely@secretlab.com, which appears to have bounced. I'm
> hoping that I just wrote the address down wrong, and the .ca
> version is correct.

Yes, the correct domain is secretlab.ca

> 
> Apologies for any confusion,
> Tibs
> 
> Begin forwarded message:
> 
> Date: Fri, 28 Jan 2011 17:50:40 +0000
> From: Tony Ibbs <tibs@kynesim.co.uk>
> To: Linux-embedded <linux-embedded@vger.kernel.org>
> Cc: Grant Likely <grant.likely@secretlab.com>, Tibs at home <tibs@tonyibbs.co.uk>, Richard Watts <rrw@kynesim.co.uk>
> Subject: [RFC][PATCH] KBUS messaging subsystem
> 
> 
[...]
> We've got a working repository with Linux 2.6.37 patched for KBUS,
> available via:
> 
>   git pull git://github.com/crazyscot/linux-2.6-kbus.git kbus-2.6.37
> 
> Our tentative patch is at
> http://kbus.googlecode.com/files/0001-KBUS-messaging-subsystem.patch
> 
> The diffstat for the patch is:
> 
>  Documentation/Kbus.txt     | 1189 ++++++++++
>  include/linux/kbus_defns.h |  666 ++++++
>  init/Kconfig               |    3 +
>  ipc/Kconfig                |   99 +
>  ipc/Makefile               |    1 +
>  ipc/kbus.c                 | 5120 ++++++++++++++++++++++++++++++++++++++++++++
>  ipc/kbus_internal.h        |  773 +++++++
>  7 files changed, 7851 insertions(+), 0 deletions(-)
> 
> Should I just submit the patch to this list?

Yes.  If you want review, then posting a URLs doesn't help.  :-)  Send
it as an email so that I can just hit 'reply' and start typing.
Post and then I'll comment.

g.

Re: [RFC][PATCH] KBUS messaging subsystem

[Tony Ibbs]

Apologies again for getting Grant's email address wrong on my initial
email.

On Tue, 1 Feb 2011 20:40:08 -0700
Grant Likely <grant.likely@secretlab.ca> wrote:

> There are already a large number of communication channels available
> to Linux, with DBUS gatting the most attention at the moment.  What
> niche is KBUS filling that DBUS and others misses?  Secondarily, does
> KBUS implement a new API instead of extending an existing mechanism to
> meet your needs?  I suspect you'll get a lot more traction if KBUS can
> make DBUS work better/faster/more efficiently.

Hmm. I'm not sure what make up that large number, but I'll try to
address your specific concerns.

I've got four real reasons for why KBUS and not something else:

1. Simple to use and understand. You should be able to get started
using KBUS without needing to feel you've had to learn anything much.

  1.5 This also means it mustn't try to do too much - it needs
  to aim for the simplest solution that will do.

2. We require deterministic message ordering. All listeners (on the
same bus) to a particular message name must receive those messages
in the same order. And that must be the same as the order they
were sent in.

3. In a request/reply situation, the requester must be guaranteed a
reply, even if the reply is to indicate that the replier has gone away,
and including the case that the replier has crashed.

4. The system has to be written in C, since we can't guarantee the
availability of other language libraries (specifically, we are quite
likely not to have C++ on a board).

I believe DBUS is ruled out because of (1) alone - it is not easy to use
or understand. Someone writing embedded code does not normally want to
invest much time in learning yet another technology, so quick to learn
is very important here. Learning about protocol layers is not
appropriate in this context. My EuroPython talk explained almost the
entirety of KBUS (and all the important bits) with examples, in about
20 minutes.

I don't know if DBUS provides (2) - I've not been able to tell from its
documentation. And I assume it doesn't provide (3).

Also, KBUS is perfectly usable with just the kernel module. DBUS is
quite large, and we don't necessarily have the space to spare for it on
many of our systems.

As a side issue, DBUS also mixes together message and content. KBUS
deliberately does not address the description of message content. We do
not believe there is a one-size-fits-all solution to this, not even for
request/reply used as RPC. Whilst not named in the big 4 above, I
regard this as a serious problem with DBUS for our domain.

Elsewhere, 0mq (zeromq) looks very attractive, not least because it
appears to be very well documented (although I think a lot of that is
fairly recent). I really want to find a project where I need to use it.
However, it is written in C++, so fails (4). Also, I think it is too
complete a solution - it is meant to be a general networking solution,
and I think that makes it wrong for this application - it is over
powerful. Also, I assume it cannot provide (3).

I'm not sure what other mechanisms we should consider. The kernel
itself doesn't provide anything that addresses our problem space,
without a lot of support work.

As to API. The basic API is just the file API (open/read/write/
ioctl/close), and KBUS is perfectly usable at that level.

For those who want to have simpler code, there's a C library to wrap
stuff up neatly, but you don't have to use it. We wanted a set of
useful routines for ourselves, so it happened anyway. It's also a
useful step to (for instance) the Java API, rather than going the
low-level route (obviously not needed when writing the C++ or Python
bindings).

Finally, I have actually submitted a CELF Open Project proposal for
KBUS, and one of the work items is to look at using KBUS as a bottom
end for DBUS.
 
> > Date: Fri, 28 Jan 2011 17:50:40 +0000
> > From: Tony Ibbs <tibs@kynesim.co.uk>
> > To: Linux-embedded <linux-embedded@vger.kernel.org>
[...]
> > Should I just submit the patch to this list?
[...]
> Yes.  If you want review, then posting a URLs doesn't help.  :-)  Send
> it as an email so that I can just hit 'reply' and start typing.
> Post and then I'll comment.

I shall attempt to sort out the issues following on from Sam Ravnborg's
comments, and then submit as several inline patches, then.

Tibs

Re: [RFC][PATCH] KBUS messaging subsystem

[Grant Likely]

On Thu, Feb 3, 2011 at 10:04 AM, Tony Ibbs <tibs@kynesim.co.uk> wrote:
> Apologies again for getting Grant's email address wrong on my initial
> email.
>
> On Tue, 1 Feb 2011 20:40:08 -0700
> Grant Likely <grant.likely@secretlab.ca> wrote:
>
>> There are already a large number of communication channels available
>> to Linux, with DBUS gatting the most attention at the moment.  What
>> niche is KBUS filling that DBUS and others misses?  Secondarily, does
>> KBUS implement a new API instead of extending an existing mechanism to
>> meet your needs?  I suspect you'll get a lot more traction if KBUS can
>> make DBUS work better/faster/more efficiently.
>
> Hmm. I'm not sure what make up that large number, but I'll try to
> address your specific concerns.
>
> I've got four real reasons for why KBUS and not something else:
>
> 1. Simple to use and understand. You should be able to get started
> using KBUS without needing to feel you've had to learn anything much.

It can be argued that this is an interface issue; but that's a
tangential point to my main point below....

[...]
> As a side issue, DBUS also mixes together message and content. KBUS
> deliberately does not address the description of message content. We do
> not believe there is a one-size-fits-all solution to this, not even for
> request/reply used as RPC. Whilst not named in the big 4 above, I
> regard this as a serious problem with DBUS for our domain.
>
> Elsewhere, 0mq (zeromq) looks very attractive, not least because it
> appears to be very well documented (although I think a lot of that is
> fairly recent). I really want to find a project where I need to use it.
> However, it is written in C++, so fails (4). Also, I think it is too
> complete a solution - it is meant to be a general networking solution,
> and I think that makes it wrong for this application - it is over
> powerful. Also, I assume it cannot provide (3).
>
> I'm not sure what other mechanisms we should consider. The kernel
> itself doesn't provide anything that addresses our problem space,
> without a lot of support work.
>
> As to API. The basic API is just the file API (open/read/write/
> ioctl/close), and KBUS is perfectly usable at that level.

The big issue that comes into play when implementing a new protocol in
Linux is that once it is implemented, we need to support it until the
end of time.  That means it needs to be well thought out and there is
a lot of resistance to implementing something new without first seeing
that it is a measurable benefit that is worth supporting to the end of
time.  In your case, you'll need to show (or attract) a wider range of
users who can say, "yes, this is better than the alternatives".  I
suggest looking at working with dbus for that exact reason.  If dbus
can be made faster/better/more reliable by using a kbus backend, then
you've got a big userbase saying that it is a valuable addition.  It
also gives you a broad use-case that will stretch your assumptions and
improve the implementation before it gets set in stone when merged.

If kbus turns out to only be applicable to your corner of the universe
then it becomes a lot harder to justify it as a generic Linux
communication implementation.

g.

Re: [RFC][PATCH] KBUS messaging subsystem

[Tony Ibbs]

On 3 Feb 2011, at 17:32, Grant Likely wrote:

> The big issue that comes into play when implementing a new protocol in
> Linux is that once it is implemented, we need to support it until the
> end of time.

Indeed.

> That means it needs to be well thought out and there is
> a lot of resistance to implementing something new without first seeing
> that it is a measurable benefit that is worth supporting to the end of
> time.

And I believe this is a good thing. If KBUS doesn't get in, hopefully it
will be because it has been demonstrated why it should not.

> In your case, you'll need to show (or attract) a wider range of
> users who can say, "yes, this is better than the alternatives".

This is one of the reasons I wanted to start by talking to the embedded
linux mailing list, where we can get the attention of people working
with small systems.

> I suggest looking at working with dbus for that exact reason.  If dbus
> can be made faster/better/more reliable by using a kbus backend, then
> you've got a big userbase saying that it is a valuable addition.

Well, that's presumably true, but does DBUS *need* to be those
things (i.e., is it doing well enough already), and is that enough
reason to put something into the kernel (previous attempts, e.g.,
http://alban.apinc.org/blog/2010/09/15/d-bus-in-the-kernel-faster,
seem to have foundered).

> It also gives you a broad use-case that will stretch your assumptions

That *is* a good motivation, and is one of the reasons for the CELF
proposals. But I don't think it is sufficient to be KBUS's reason for
existence.

> and improve the implementation before it gets set in stone when
> merged.

and I'm not so convinced on this one, since it seems to present a bit of
a chiken-and-egg problem. We believe KBUS is ready for merging because
we've been using it to good effect. If we don't try to merge it, then we
won't find out if we're right (!). If it should be in the kernel, then
it's better to get it there sooner, so that it can be moulded/grown
in-tree.

> If kbus turns out to only be applicable to your corner of the universe
> then it becomes a lot harder to justify it as a generic Linux
> communication implementation.

On the other hand, that depends on how big the corner is. We've seen
many projects trying to hand construct new socket based systems for lack
of a KBUS, and these generally reinvent the same problems each time. The
idea of KBUS is to get away from that.

If you're working on a large-ish system, needing to interact with the
many high-level components that already interact using DBUS, then
clearly you should be using DBUS for that purpose.

However, if you're building a system from scratch, if you have Linux,
busybox, and a set of programs that need to communicate with each other,
and limited resources, then DBUS is not going to be chosen. But those
systems still need a reliable, already written messaging mechanism,
rather than another ad-hoc approach.

Personally, I don't see KBUS competing with DBUS. DBUS already has its
place, and I assume it is good at what it does. KBUS wants to come in
at a lower level, and satisfy different customers.

Tibs

[RFC][PATCH] KBUS messaging subsystem

[Tony Ibbs]

KBUS is a lightweight Linux-kernel mediated message system,
particularly intended for use in embedded environments.

It is meant to be simple to use and understand. It is particularly
designed to provide predictable message delivery, deterministic
message ordering, and a guaranteed reply for each request.

We have various customers using KBUS in real life, and believe it to
be useful. I had a showcase table for KBUS at the ELCE in Cambridge,
October last year, and there seemed to be interest.

We'd like some advice on how to submit this - I assume that submitting
patches to the Embedded Linux mailing list first would be sensible,
since we believe this to be an area that will find KBUS interesting.

The KBUS project home page is at http://kbus-messaging.org/
The KBUS specification can be found at
http://html.kbus.googlecode.com/hg/docs/html/specification.html

Userland support, plus unit tests for the kernel module, are in the
Google code repositories. There's a GettingStarted page linked from
the KBUS home page.

We've got a working repository with Linux 2.6.37 patched for KBUS,
available via:

  git pull git://github.com/crazyscot/linux-2.6-kbus.git kbus-2.6.37

Our tentative patch is at
http://kbus.googlecode.com/files/0001-KBUS-messaging-subsystem.patch

The diffstat for the patch is:

 Documentation/Kbus.txt     | 1189 ++++++++++
 include/linux/kbus_defns.h |  666 ++++++
 init/Kconfig               |    3 +
 ipc/Kconfig                |   99 +
 ipc/Makefile               |    1 +
 ipc/kbus.c                 | 5120 ++++++++++++++++++++++++++++++++++++++++++++
 ipc/kbus_internal.h        |  773 +++++++
 7 files changed, 7851 insertions(+), 0 deletions(-)

Should I just submit the patch to this list?

Tibs

Re: [RFC][PATCH] KBUS messaging subsystem

[Sam Ravnborg]

[Edited cc: list as my smtp server refused to send the original reply]
> 
> Our tentative patch is at
> http://kbus.googlecode.com/files/0001-KBUS-messaging-subsystem.patch
> 
> The diffstat for the patch is:
> 
>  Documentation/Kbus.txt     | 1189 ++++++++++
>  include/linux/kbus_defns.h |  666 ++++++
>  init/Kconfig               |    3 +
>  ipc/Kconfig                |   99 +
>  ipc/Makefile               |    1 +
>  ipc/kbus.c                 | 5120 ++++++++++++++++++++++++++++++++++++++++++++
>  ipc/kbus_internal.h        |  773 +++++++
>  7 files changed, 7851 insertions(+), 0 deletions(-)
> 
> Should I just submit the patch to this list?

You will get more feedback on lkml...
Looks like comprehensive descriptions - good!

Took a quick peek at the patch.

- use __u32 (and friends) in exported headers - not uint32_t.
- do not invent our own logging system. Use one of the standard methods.

If you post for review try to divide it up a little.
Maybe kbus.c in one patch, Makefile + Kconfig in last patch.
It is a large file to review already.

	Sam

Re: [RFC][PATCH] KBUS messaging subsystem

[Tony Ibbs]

On Fri, 28 Jan 2011 20:20:42 +0100
Sam Ravnborg <sam@ravnborg.org> wrote:

> [Edited cc: list as my smtp server refused to send the original reply]

Apologies - I had copied down Grant Likely's email address wrong at ELCE.

> > Should I just submit the patch to this list?
> 
> You will get more feedback on lkml...

Indeed. I hoped to start small.

> - use __u32 (and friends) in exported headers - not uint32_t.

OK. I'd missed that. Presumably I should also be using u32 and friends
inside the kernel module itself, instead of uint32_t...

> - do not invent our own logging system. Use one of the standard
> methods.

I vaguely remember having a reason for that. Doubtless it was a bad
one. I shall attend to it.

> If you post for review try to divide it up a little.
> Maybe kbus.c in one patch, Makefile + Kconfig in last patch.
> It is a large file to review already.

That's useful guidance. I shall amend the code as suggested above and
resubmit as smaller, inline patches.

Tibs