use libqmi or .json definition for qmi calls

use libqmi or .json definition for qmi calls

[Alexander Couzens]

[-- Attachment #1: Type: text/plain, Size: 633 bytes --]

Hi,

as it get's quite exhausting to create over and over qmi messages by
hand, I'm wondered why ofono isn't using a library for it.
Is there a reason why ofono isn't using libqmi [1] or the .json [2] to
generate code like other projects does [3]?

Best,
lynxis

[1] https://www.freedesktop.org/wiki/Software/libqmi/
[2] https://cgit.freedesktop.org/libqmi/tree/data/qmi-service-wms.json
[3] https://git.lede-project.org/?p=project/uqmi.git;a=tree;f=data;hb=HEAD
-- 
Alexander Couzens

mail: lynxis(a)fe80.eu
jabber: lynxis(a)fe80.eu
mobile: +4915123277221
gpg: 390D CF78 8BF9 AA50 4F8F  F1E2 C29E 9DA6 A0DF 8604

[-- Attachment #2: attachment.sig --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: use libqmi or .json definition for qmi calls

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 1055 bytes --]

Hi Alexander,

On 05/16/2017 11:14 AM, Alexander Couzens wrote:
> Hi,
>
> as it get's quite exhausting to create over and over qmi messages by
> hand, I'm wondered why ofono isn't using a library for it.
> Is there a reason why ofono isn't using libqmi [1] or the .json [2] to
> generate code like other projects does [3]?
>

libqmi is quite large, depends on weird glib features and is really not 
an option for us.  Furthermore, we will eventually be migrating away 
from glib, so introducing additional dependencies or libraries that 
depend on glib will just make our life harder in the future.

When we started our QMI driver, libqmi was still in its early stages. 
Much of the work understanding the protocol still had to be done by 
hand.  Back then we decided that auto-generating the code wasn't 
worthwhile.  It is almost certain that there will not be multiple QMI 
based drivers.  So by the time you wrote the JSON definition or 
equivalent, you could have implemented the driver method that used it.

Regards,
-Denis

Re: use libqmi or .json definition for qmi calls

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 2154 bytes --]

Hi Denis,

On Tue, May 16, 2017 at 01:33:56PM -0500, Denis Kenzior wrote:
> libqmi is quite large, depends on weird glib features and is really not an
> option for us.  Furthermore, we will eventually be migrating away from glib,
> so introducing additional dependencies or libraries that depend on glib will
> just make our life harder in the future.

I can follow that argument.  However, it only applies to libqmi, and not
to uqmi, which doesn't use glib.  On the other hand, it appears to be
using libubox, which is very OpenWRT/LEDE specific.

> When we started our QMI driver, libqmi was still in its early stages. Much
> of the work understanding the protocol still had to be done by hand.  Back
> then we decided that auto-generating the code wasn't worthwhile.  It is
> almost certain that there will not be multiple QMI based drivers.  So by the
> time you wrote the JSON definition or equivalent, you could have implemented
> the driver method that used it.

This explains the history, sure.

However, it would still be good to get an opinion on what your thoughts
are to use code generation from the JSON file instead?  The situation
today is different:  There is a JSON File that appears to have more
complete QMI coverage than ofono, so it's either "manually writing code
to catch up with that file" or "be open towards generation of ofono QMI
code from that JSON file"

Let's assume somebody would work on generating code from the JSON file
that had no dependencies to "weird glib features" or glib at all, or a
library like libubox:
Would you think it as worth-while to merge such a QMI driver, either as
separate driver, or even as a replacement to the current one?

The amount of interfaces, interface versions and messages on QMI is only
going to increase over time.

Regards,
	Harald
-- 
- Harald Welte <laforge@gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

Re: use libqmi or .json definition for qmi calls

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 3045 bytes --]

Hi Harald,

On 05/22/2017 03:54 AM, Harald Welte wrote:
> Hi Denis,
>
> On Tue, May 16, 2017 at 01:33:56PM -0500, Denis Kenzior wrote:
>> libqmi is quite large, depends on weird glib features and is really not an
>> option for us.  Furthermore, we will eventually be migrating away from glib,
>> so introducing additional dependencies or libraries that depend on glib will
>> just make our life harder in the future.
>
> I can follow that argument.  However, it only applies to libqmi, and not
> to uqmi, which doesn't use glib.  On the other hand, it appears to be
> using libubox, which is very OpenWRT/LEDE specific.

Didn't know uqmi existed.  Looks like that project started about a year 
after qmi driver was functional.  So again, wasn't really a consideration.

But just to make it clear: anything we introduce would have to depend 
only on libc and perhaps on the bare minimum set of features we use from 
glib at the moment.

>
>> When we started our QMI driver, libqmi was still in its early stages. Much
>> of the work understanding the protocol still had to be done by hand.  Back
>> then we decided that auto-generating the code wasn't worthwhile.  It is
>> almost certain that there will not be multiple QMI based drivers.  So by the
>> time you wrote the JSON definition or equivalent, you could have implemented
>> the driver method that used it.
>
> This explains the history, sure.
>
> However, it would still be good to get an opinion on what your thoughts
> are to use code generation from the JSON file instead?  The situation
> today is different:  There is a JSON File that appears to have more
> complete QMI coverage than ofono, so it's either "manually writing code

complete, sure.  But then we only use a very small subset of QMI. 
Pointless to generate all this code if you use 5-10% of it.

> to catch up with that file" or "be open towards generation of ofono QMI
> code from that JSON file"
>

Generated code is rarely pretty, or maintainable.  The code generator 
itself can also be quite large.  We played with the idea initially, but 
didn't think the cost-benefit was there.

I have seen auto-generated code used to good effect (e.g. in the Linux 
kernel), but the rules are arcane and bugs easily introduced / 
overlooked.  So I'm open minded, but the devil will be in the details.

> Let's assume somebody would work on generating code from the JSON file
> that had no dependencies to "weird glib features" or glib at all, or a
> library like libubox:
> Would you think it as worth-while to merge such a QMI driver, either as
> separate driver, or even as a replacement to the current one?

I can understand auto-generating qmi data structures and parsers / 
builders, but how are you going to auto-generate the driver?

>
> The amount of interfaces, interface versions and messages on QMI is only
> going to increase over time.
>

You'll have to educate me on that, so far it seems QMI has been frozen 
for 5+ years.

Regards,
-Denis

Re: use libqmi or .json definition for qmi calls

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 5501 bytes --]

Hi Denis,

On Mon, May 22, 2017 at 10:01:03AM -0500, Denis Kenzior wrote:
> > I can follow that argument.  However, it only applies to libqmi, and not
> > to uqmi, which doesn't use glib.  On the other hand, it appears to be
> > using libubox, which is very OpenWRT/LEDE specific.
> 
> Didn't know uqmi existed.  Looks like that project started about a year
> after qmi driver was functional.  So again, wasn't really a consideration.

Nobody argues that things should or could have done different in the
past. My comments are strictly reserved to the future.

Does it make sense to extend the hand-written QMI encoder/decoder code,
or does it make more sense to invest time in re-using the apparently
more complete description [and work] in the libqmi+uqmi world?

> But just to make it clear: anything we introduce would have to depend only
> on libc and perhaps on the bare minimum set of features we use from glib at
> the moment.

That's perfectly fine and I don't think it would be an issue.

> complete, sure.  But then we only use a very small subset of QMI. Pointless
> to generate all this code if you use 5-10% of it.

As far as I know, the ofono qmi driver does not cover yet all the ofono
dbus interfaces that are documented (at least not for all the QMI
versions seen out there?).  So to get 100% of the ofono interfaces
implemented, you probably will need to implement more of QMI.

Also, with modems/phones implementing things like IMS for LTE (and WiFi
calling), isn't it likely that ofono will also have to cover/implement
related parts of QMI that deal with that?

To me it seemed ofono always wanted to have a more feature-complete
code base to talk to various modems, well beyond the basic voice call +
sms capabilities offered by other projects (openmoko gsmd,
freesmartphone.org) or the single-purpose data-only ModemManager..  I
mean, why else are there topics like SIM toolkit, call hold,
supplementary services, etc. covered in the mission statement and some
of the code?

> Generated code is rarely pretty, or maintainable.  

it is not supposed to be.  You are editing the templates in the
generator, not the generated code...

> The code generator itself can also be quite large.  We played with the
> idea initially, but didn't think the cost-benefit was there.

The benefit might not be there for you alone, but the issue is that
there are different FOSS QMI implementations out there (libqmi, uqmi,
ofono and possibly more), and implementing encoding and parsing of each
new message or interface N number of times by hand in each project seems
rather inefficient use of the constantly limited resources that FOSS
projects seem to have.

> > Would you think it as worth-while to merge such a QMI driver, either as
> > separate driver, or even as a replacement to the current one?
> 
> I can understand auto-generating qmi data structures and parsers / builders,
> but how are you going to auto-generate the driver?

the driver of course has to be written by hand.  But not the
encoding/decoding of routines.

> You'll have to educate me on that, so far it seems QMI has been frozen for
> 5+ years.

What are you referring to specifically? The QMI you see in modems out
there? The ofono implementation of it? The specific sub-set used by
ofono so far?

For the sake of an argument: It might even be that the modem/baseband
side is fixed for the past 5 years.  But does ofono implement all
"relevant" features that were present in QMI modems 5 years ago in its
qmimodem driver?

"relevant" to me is pretty much anything that can be done from a phone
side, i.e. anything related to
* device identification + control (on/off/airplane)
* network selection (scanning, manual selection, ...)
* voice call including call hold / multiple calls, dtmf, supplementary services
* SMS MO + MT, including delivery reports, *, setting SMSC, ...
* USSD
* activation/deactivation of PDP contexts [including multiple parallel]

We're using ofono from osmo-gsm-tester (see
ftp://ftp.osmocom.org/docs/latest/osmo-gsm-tester-manual.pdf and
http://git.osmocom.org/osmo-gsm-tester/) in order to do full end-to-end
tests on the Osmocom GSM stack including OsmoBTS, OsmoNITB, etc.

From the what the team working on this at sysmocom (Pau Espin, Alexander
Couzens, formerly myself...) have seen so far, it is very hard to have
even only a subset of the above work reliably on any given modem tested
so far.  We originally started with some AT-command based Sierra
Wireless modems (and gave up on that about a year ago) and have since
re-build the tester hardware towards using QMI-based modems.

We've been looking at modern LTE-capable devices like Quectel EC-20,
EC-25, Sierra Wireless MC7304 or even older devices like Gobi2000
and Gobi3000.

I am not complaining here.  I just want to figure out what is the best
strategy to get ofono to work reliably for all "relevant" features on
initially at least one QMI based modem, later on hopefully with many
other modems [using different baseband chipsets and stacks, to test
interoperability of our stack].

Regards,
	Harald
-- 
- Harald Welte <laforge@gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

Re: use libqmi or .json definition for qmi calls

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 5934 bytes --]

Hi Harald,

>
> Does it make sense to extend the hand-written QMI encoder/decoder code,
> or does it make more sense to invest time in re-using the apparently
> more complete description [and work] in the libqmi+uqmi world?
>

Or simply use the JSON description as an aid to generate the 
'hand-written' code.

>> complete, sure.  But then we only use a very small subset of QMI. Pointless
>> to generate all this code if you use 5-10% of it.
>
> As far as I know, the ofono qmi driver does not cover yet all the ofono
> dbus interfaces that are documented (at least not for all the QMI
> versions seen out there?).  So to get 100% of the ofono interfaces
> implemented, you probably will need to implement more of QMI.
>

It is probably not complete, but it actually does cover quite a bit of 
the cases.  I remember looking at the QMI specs long ago and thinking 
that we wouldn't use even half of the stuff in there.  Think of it this 
way, AT modems come with hundreds of AT commands.  We only use a fairly 
small percentage of them to implement our drivers.

> Also, with modems/phones implementing things like IMS for LTE (and WiFi
> calling), isn't it likely that ofono will also have to cover/implement
> related parts of QMI that deal with that?
>

So in theory yes.  But in practice do such devices actually exist?  When 
we made QMI work, only data cards were available.

> To me it seemed ofono always wanted to have a more feature-complete
> code base to talk to various modems, well beyond the basic voice call +
> sms capabilities offered by other projects (openmoko gsmd,
> freesmartphone.org) or the single-purpose data-only ModemManager..  I
> mean, why else are there topics like SIM toolkit, call hold,
> supplementary services, etc. covered in the mission statement and some
> of the code?
>

So yes, oFono implements a bunch of these features that other projects 
do not have.  But you do need actual hardware/firmware that supports 
these features and documentation on how these features are implemented 
on that hardware/firmware.

In practice consumer-level devices never exposed this stuff.  So there 
has always been great disparity in terms of what oFono could do and what 
it was able to do on in a particular driver/device.  It is now getting 
better with vendors such as Telit, uBlox, etc which is encouraging.

>> The code generator itself can also be quite large.  We played with the
>> idea initially, but didn't think the cost-benefit was there.
>
> The benefit might not be there for you alone, but the issue is that
> there are different FOSS QMI implementations out there (libqmi, uqmi,
> ofono and possibly more), and implementing encoding and parsing of each
> new message or interface N number of times by hand in each project seems
> rather inefficient use of the constantly limited resources that FOSS
> projects seem to have.
>

So again, I don't disagree.  But ultimately, each project should do what 
is right for that project.

>> You'll have to educate me on that, so far it seems QMI has been frozen for
>> 5+ years.
>
> What are you referring to specifically? The QMI you see in modems out
> there? The ofono implementation of it? The specific sub-set used by
> ofono so far?

The first one.

>
> For the sake of an argument: It might even be that the modem/baseband
> side is fixed for the past 5 years.  But does ofono implement all
> "relevant" features that were present in QMI modems 5 years ago in its
> qmimodem driver?
>

No, not all of it.  We got enough working for our particular project. 
We have not developed it actively since, as you can probably tell.

> "relevant" to me is pretty much anything that can be done from a phone
> side, i.e. anything related to

So off the top of my head here:

> * device identification + control (on/off/airplane)

yes

> * network selection (scanning, manual selection, ...)

yes

> * voice call including call hold / multiple calls, dtmf, supplementary services

no.  Don't think there is a device that supports voice calls anyhow

> * SMS MO + MT, including delivery reports, *, setting SMSC, ...

yes

> * USSD

no

> * activation/deactivation of PDP contexts [including multiple parallel]

yes, but AFAIK no qmi device supports multiple of these.

>
> We're using ofono from osmo-gsm-tester (see
> ftp://ftp.osmocom.org/docs/latest/osmo-gsm-tester-manual.pdf and
> http://git.osmocom.org/osmo-gsm-tester/) in order to do full end-to-end
> tests on the Osmocom GSM stack including OsmoBTS, OsmoNITB, etc.
>
> From the what the team working on this at sysmocom (Pau Espin, Alexander
> Couzens, formerly myself...) have seen so far, it is very hard to have
> even only a subset of the above work reliably on any given modem tested
> so far.  We originally started with some AT-command based Sierra
> Wireless modems (and gave up on that about a year ago) and have since
> re-build the tester hardware towards using QMI-based modems.
>
> We've been looking at modern LTE-capable devices like Quectel EC-20,
> EC-25, Sierra Wireless MC7304 or even older devices like Gobi2000
> and Gobi3000.
>
> I am not complaining here.  I just want to figure out what is the best
> strategy to get ofono to work reliably for all "relevant" features on
> initially at least one QMI based modem, later on hopefully with many
> other modems [using different baseband chipsets and stacks, to test
> interoperability of our stack].
>

Why are you using QMI then?  I would recommend using something that is 
actually meant to be fully featured.  E.g. pass full blown GCF (not just 
the limited subset for data-only devices.)  Something that comes with 
developer documentation, so that if a feature is not supported in the 
current oFono driver, it is easy to add.  Look at Telit or uBlox.

Regards,
-Denis

Re: use libqmi or .json definition for qmi calls

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 7673 bytes --]

Hi Denis,

On Mon, May 22, 2017 at 12:37:35PM -0500, Denis Kenzior wrote:
> > Does it make sense to extend the hand-written QMI encoder/decoder code,
> > or does it make more sense to invest time in re-using the apparently
> > more complete description [and work] in the libqmi+uqmi world?
> 
> Or simply use the JSON description as an aid to generate the 'hand-written'
> code.

indeed, an option.

> > Also, with modems/phones implementing things like IMS for LTE (and WiFi
> > calling), isn't it likely that ofono will also have to cover/implement
> > related parts of QMI that deal with that?
> 
> So in theory yes.  But in practice do such devices actually exist?  When we
> made QMI work, only data cards were available.

Yes, they do exist.  Based on my reverse-engineering efforts on the
EC-20 / EC-25, I can definitely confirm this.  And they are using quite
old base versions from Qualcomm..

Actually, quite a lot of the complicated software architecture on those
more modern 4G-enabled Qualcomm MDM chipsets seems to be a result of the
support for features like wifi-offloading and voice call support,
including SRVCC across different RAN technologies (2G/3G/4G/WiFi).  You
should see the amount of alsa controls exposed on the Linux
inside those modems (that's the voice side, but just for illustration).

As Sierra Wireless are also using the Linux-based stack for the modems,
I would assume they have related LTE voice capability, too.

> So yes, oFono implements a bunch of these features that other projects do
> not have.  But you do need actual hardware/firmware that supports these
> features and documentation on how these features are implemented on that
> hardware/firmware.

In theory, yes.  In practise, you can do a lot without documentation
simply based on the information that can be found in from leaked QMI
source code in various places.  Basically just like with any other piece
of hardware/firmware that's supported by FOSS without any help from its
vendor.

> In practice consumer-level devices never exposed this stuff.  So there has
> always been great disparity in terms of what oFono could do and what it was
> able to do on in a particular driver/device.  It is now getting better with
> vendors such as Telit, uBlox, etc which is encouraging.

I agree for AT-based modems. But I think this is where for Qualcomm
based devices QMI makes a big difference: The individual modem maker
would actively have to disrupt it from working e.g. by removing parts of
the QMI implementation / interfaces in the firmware..

> > * network selection (scanning, manual selection, ...)
> yes

was not working out-of-the-box with the qmi driver,

> > * voice call including call hold / multiple calls, dtmf, supplementary services
> 
> no.  Don't think there is a device that supports voice calls anyhow

Of course there are plenty!  Even with documented voice features from
various manufacturers.  In some cases you can buy variants with and
without voice functionality (probably related to different patent fees
whether or not voice support is included).

You can find modems that
* expose voice only on analog mic/speaker pads, that's typically the
  case for solder/lga-type modems, whether from Gemalto, Telit, Simcom,
  Quectel, u-blox or virtually any other vendor.  Voice signalling is
  then done via AT commands or QMI.
* expose voice via PCM bus on some otherwise unused non-standard pins of
  the mPCIe.  This is the case for some Huawei and Sierra Wireless
  devices, and the reason why we built
  http://osmocom.org/projects/mpcie-breakout/wiki
* expose voice via USB-Audio profile (possible with a lot of modern
  Sierra Wireless modems in mPCIe or M.2 form-factor)
* expose voice via some proprietary "GSM codec frame over USB Serial"
  which is ugly but better than having to attach a codec IC to the PCM
  bus.  Seen in some older 2G/3G-only quectel devices, for example.

And hey, even the Openmoko phones (which were/are supported by ofono,
AFAIR) had voice support more than a decade ago.  Not with QMI, though.

> > * activation/deactivation of PDP contexts [including multiple parallel]
> 
> yes, but AFAIK no qmi device supports multiple of these.

ok, we'll have to look at this.  at least with LTE modems, it is the
normal operation to have multiple EPS bearers active in parallel.  At
least one for signalling and one for data, maybe more for voice, etc.

> > I am not complaining here.  I just want to figure out what is the best
> > strategy to get ofono to work reliably for all "relevant" features on
> > initially at least one QMI based modem, later on hopefully with many
> > other modems [using different baseband chipsets and stacks, to test
> > interoperability of our stack].
> 
> Why are you using QMI then?  

* Because AT command suck badly. Every vendor gets it wrong in a subtly
  different way, and every modem vendor has it's own incompatible
  extensions to it
* QMI is something present in almost all devices based on Qualcomm,
  which is the largest supplier of baseband chips for quite some time.
  So by having QMI supported well, we unlikely will have to worry about
  differences between a Quectel or a Sierra Wireless device.  If we use
  AT commands, we do.
* QMI is much easier and more natural to parse in a program than the
  string-based AT commands, particularly thinking of mixture of
  unsolicited results codes vs, request-response style commands, the
  issue of voice related commands "blocking" an AT command channel,
  having to use TS 07.10 multiplex, etc.

> I would recommend using something that is actually meant to be fully
> featured.  E.g. pass full blown GCF (not just the limited subset for
> data-only devices.)  

Where is the list of current hardware that is supported best by ofono?
the information contained in its documentation seems like it's a decade
(I may exaggerate) old and not maintained.

Why do you claim that the devices we currently use (or evaluate to use)
are data-only?

> Something that comes with developer documentation, so that if a
> feature is not supported in the current oFono driver, it is easy to
> add.  Look at Telit or uBlox.

There is no shortage in terms of "developer" style documentation from
vendors like Sierra Wireless, Huawei and Quectel.  More recently also
ZTE. In fact, you would be surprised at the level of detail that some of
them have.  The ZTE documentation even goes as low as to show you the
message sequence chart of what happens inside the GSM radio and core
network for each singla AT command you issue towards it.

I'm not sure in which way Telit or u-blox should differ here, except
maybe that they u-blox (like Gemalto, I don't know bout Telit)
artificially lock out their users from using the diagnostic features of
modems, so the use of things like osmo-qcdiag to get protocol traces for
debugging is made impossible.

So the problem is not really to know how to make a modem perform a given
function.  The problem is how to abstract away the difference between
many different modems (possibly even based on different chipsets/stack
vendors) without having to start from scratch.  And that's exactly what
I thought ofono was created for :)

Regards,
	Harald
-- 
- Harald Welte <laforge@gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

Re: use libqmi or .json definition for qmi calls

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 6117 bytes --]

Hi Harald,
> 
>>> * voice call including call hold / multiple calls, dtmf, supplementary services
>> 
>> no.  Don't think there is a device that supports voice calls anyhow
> 
> Of course there are plenty!  Even with documented voice features from
> various manufacturers.  In some cases you can buy variants with and
> without voice functionality (probably related to different patent fees
> whether or not voice support is included).
> 
> You can find modems that
> * expose voice only on analog mic/speaker pads, that's typically the
>  case for solder/lga-type modems, whether from Gemalto, Telit, Simcom,
>  Quectel, u-blox or virtually any other vendor.  Voice signalling is
>  then done via AT commands or QMI.
> * expose voice via PCM bus on some otherwise unused non-standard pins of
>  the mPCIe.  This is the case for some Huawei and Sierra Wireless
>  devices, and the reason why we built
>  http://osmocom.org/projects/mpcie-breakout/wiki
> * expose voice via USB-Audio profile (possible with a lot of modern
>  Sierra Wireless modems in mPCIe or M.2 form-factor)
> * expose voice via some proprietary "GSM codec frame over USB Serial"
>  which is ugly but better than having to attach a codec IC to the PCM
>  bus.  Seen in some older 2G/3G-only quectel devices, for example.
> 
> And hey, even the Openmoko phones (which were/are supported by ofono,
> AFAIR) had voice support more than a decade ago.  Not with QMI, though.

I’m confused.  Are you talking about voice capable modems in general or the QMI ones?  I’m of course aware of various Huawei, etc modems that support voice via AT commands, RIL, ISI, etc.  I’m not aware of any QMI based devices.  QMI protocol does support voice, so I’m sure they exist.  Just not aware of any.  That is why we never bothered to write the voicecall atom driver for QMI.  If you want to contribute that, I’m happy to take it.

> 
>>> * activation/deactivation of PDP contexts [including multiple parallel]
>> 
>> yes, but AFAIK no qmi device supports multiple of these.
> 
> ok, we'll have to look at this.  at least with LTE modems, it is the
> normal operation to have multiple EPS bearers active in parallel.  At
> least one for signalling and one for data, maybe more for voice, etc.

Yes, but at least the gobi cards I’ve seen only expose a single network interface.  So multiple contexts are not possible.

> 
> * Because AT command suck badly. Every vendor gets it wrong in a subtly
>  different way, and every modem vendor has it's own incompatible
>  extensions to it

I agree there, but it is not really that bad any more.

> * QMI is something present in almost all devices based on Qualcomm,
>  which is the largest supplier of baseband chips for quite some time.
>  So by having QMI supported well, we unlikely will have to worry about
>  differences between a Quectel or a Sierra Wireless device.  If we use
>  AT commands, we do.

Fair enough, but you have to make a working QMI driver first.  QMI has its own share of idiosyncrasies. 

> * QMI is much easier and more natural to parse in a program than the
>  string-based AT commands, particularly thinking of mixture of
>  unsolicited results codes vs, request-response style commands, the
>  issue of voice related commands "blocking" an AT command channel,
>  having to use TS 07.10 multiplex, etc.

There are very few AT commands that block, and oFono solves the multiplexing, unsolicited results, etc for you.  I think you’re overstating the issue, but I do not disagree with you.

> 
>> I would recommend using something that is actually meant to be fully
>> featured.  E.g. pass full blown GCF (not just the limited subset for
>> data-only devices.)  
> 
> Where is the list of current hardware that is supported best by ofono?
> the information contained in its documentation seems like it's a decade
> (I may exaggerate) old and not maintained.

No such documentation exists.  We had to pick and choose our battles and thus never had the resources for something like that.  I’m happy to host such a matrix in case someone is interested in contributing it.

> 
> Why do you claim that the devices we currently use (or evaluate to use)
> are data-only?

This was not my intent.  You’re using ofono + QMI to auto test your base station software.  Yet our driver doesn’t support voice calls.  And I’ve never seen a voice capable QMI device, though they probably exist.

Regardless, this isn’t really important.  If you’re set on QMI, then its not my place to discourage you. 

> 
>> Something that comes with developer documentation, so that if a
>> feature is not supported in the current oFono driver, it is easy to
>> add.  Look at Telit or uBlox.
> 
> There is no shortage in terms of "developer" style documentation from
> vendors like Sierra Wireless, Huawei and Quectel.  More recently also
> ZTE. In fact, you would be surprised at the level of detail that some of
> them have.  The ZTE documentation even goes as low as to show you the
> message sequence chart of what happens inside the GSM radio and core
> network for each singla AT command you issue towards it.
> 
> I'm not sure in which way Telit or u-blox should differ here, except
> maybe that they u-blox (like Gemalto, I don't know bout Telit)
> artificially lock out their users from using the diagnostic features of
> modems, so the use of things like osmo-qcdiag to get protocol traces for
> debugging is made impossible.
> 
> So the problem is not really to know how to make a modem perform a given
> function.  The problem is how to abstract away the difference between
> many different modems (possibly even based on different chipsets/stack
> vendors) without having to start from scratch.  And that's exactly what
> I thought ofono was created for :)
> 

Agreed.  But then I’m a bit confused on what you’re trying to accomplish?  If you want to enable more QMI features in oFono, then I’m all for you doing that.  Contributions are always welcome.

Regards,
-Denis

Re: use libqmi or .json definition for qmi calls

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 7912 bytes --]

Hi Denis,

On Mon, May 22, 2017 at 04:03:16PM -0500, Denis Kenzior wrote:
> I’m confused.  Are you talking about voice capable modems in general
> or the QMI ones?  

Both.  I may have misinterpreted your statement in general towards
modems.

> I’m of course aware of various Huawei, etc modems that support voice
> via AT commands, RIL, ISI, etc.  

Ok, sorry.

> I’m not aware of any QMI based devices.  QMI protocol does support
> voice, so I’m sure they exist.  Just not aware of any.  That is why we
> never bothered to write the voicecall atom driver for QMI.  If you
> want to contribute that, I’m happy to take it.

On the one hand, I believe virtually any Qualcomm(Android?) based
smartphone is using QMI to control voice.  It might be hidden from the
normal user behind rild, an it might be that virtuall nobody is using
ofono on those devices.

But then, as I iterated in my previous mail, there are plenty of modems
in solder type (LGA), mPCIe and M.2/NGFF form-factor that expose voice
capabilities via QMI.  There's a bunch of Sierra Wireless ones (like
MC7304 & co) as well as EC-20 and EC-25 from Quectel.  I also have some
Hauwei branded here that do QMI for voice.

And yes, we will in some way or another interface QMI for voice on at
least a subset of those modems.  The question is whether we spend time
on hand-crafting messages [error prone] or use the existing JSON
description and generated parser/encoders.  I'm not sure which way is
faster/easier.  If you are fundamentally against code genreation (which
you are not, AFAICT), we would skip that option.  But if you're open to
it within the constraints regarding glib and external dependencies, it
might be an idea to try.

Having spent close to a decade of my life implementing GSM protocols
from hand vs. implementing some using code generation, I am very much in
favor of the latter, whenever possible.

> Yes, but at least the gobi cards I’ve seen only expose a single
> network interface.  So multiple contexts are not possible.

I think this has all slightly changed a bit in the last 3+ years when
the Qualcomm LE stack is used on modern MDM chips.  There you have a
Linux kernel (and adb, but real glibc and no android userland) inside a
Cortex-A5 on the modem itself.  In those devices, you can at runtime
change the configuration/coposition of the USB interface and expose many
more things that are otherwise not exposed.

Basically, on those modern Qualcomm modems, the USB device you're
talking to from the ofono side is not the baseband processor, but it is
a composite android Linux kernel USB gadget.  See the 33C3 talk Holger
and I gave, as well as the information collected on
http://osmocom.org/projects/quectel-modems/wiki

We are not there yet, but after voice, USSD and all the classic features
we'll definitely look into multiple PDP contexts / EPS bearers.

Also, some modern Qualcomm chipsets also seem to have an "IPA" (Internet
Packet Accelerator) which can filter the packets in hardware.  I haven't
fully analyzed it yet (hard without documentation) but it might be that
this can be used to filter devices from one "USB network interface" into
multiple queues / bearers / contexts on the baseband side.  Still
subject to further study.  The Qualcomm kernel patches are only 2
million lines of code, AFAIR ;)

> No such documentation exists.  We had to pick and choose our battles
> and thus never had the resources for something like that.  I’m happy
> to host such a matrix in case someone is interested in contributing
> it.

Well, hosting/maintaining the document (which could just be in git) is
the easy part.  The more difficult part is collecting information of who
has actually successfully used which feature on which modem using which
modem firmware version and which ofono version.

In the best case, we are able to provide such information based on the
osmo-gsm-tester in the future.  Our main use case is to test a fixed
and trusted combination of (ofono + modem + modem firmware) against our
changing GSM/GPRS/UTMS cellular network stack.  But the same setup (or a
copy of it) could of course also be used to have a fixed version of the
GSM/GPRS/UMTS network and test a given [new] ofono version/commit
against one or multiple modems as part of continuous automation.

So I think there's something for both projects to benefit from.

But before we can do any of the above, we first have to establish a
reliably working base line configuration.

> Regardless, this isn’t really important.  If you’re set on QMI, then
> its not my place to discourage you. 

Well, I hear you. But for now we will continue down the QMI path for
stated reasons.  The question is probably more whether we can get it to
work inside ofono the way we want/need, or whether we have to consider
alternatives like using libqmi or uqmi directly.  I don't like the
latter part, because eventually we will want to talk to non-QMI /
non-Qualcomm modems, and then we need our own abstraction across those
different modems - which is the same purpose as ofono has, at least from
my point of view.

> Agreed.  But then I’m a bit confused on what you’re trying to
> accomplish?  

We want to test our GSM/GPRS/UMTS network side protocol stack and
network elements, preferably with as many different modems as
possible, but starting with Qualcomm modems with QMI and DIAG as that
gives us additional insight into what's happening inside the modem-side
protocol stack.

> If you want to enable more QMI features in oFono, then
> I’m all for you doing that.  Contributions are always welcome.

We have already started to contribute (see some smaller patches by
Alexander Couzens and Pau Espin), and there will almost certainly be
more.  But before we attempt to create some new QMI driver based on
generated code, it was worth to discuss whether or not there is any
chance for that to be merged at all.

So we want to (and will!) contribute, in terms of bug reports, patches
as well as testing.

Trying to use ofono in this context with several different modems for
more than a year now (on and off) it just seems quite a lot harder than
I would have ever anticipated.  In order to enable quicker adoption, I
strongly suggest to create a list of tested/supported modems (+ modem fw
versions) that people can start with, if they want to get the best
out-of-the-box experience.

That list doesn't have to include 100 modems.  So even if it's two
models only on that list, this is more useful than outdated documents
indicating what devices (no longer available today) worked 10 years ago.

It is of course your own discretion to do as you please in ofono.
Please don't treat this as a complaint, but just a kind suggestion from
a friend.  I would argue that if somebody with my combined telecom +
Linux / FOSS background and knowledge has serious issues to get things
to work, then many other people with less knowledge in either or both of
the fields must have given up over time.

Let me re-iterate that question: Which modem should I buy today for the
most feature-complete + reliable ofono support?  Please specify an exact
model number and preferably firmware version.  Then we can buy one of
these and compare the results of that "known good" vs. whatever we're
trying on other modems.  That also helps to differentiate while
debugging between what might be a modem/driver specific issue and a more
general ofono issue.

Regards,
	Harald

-- 
- Harald Welte <laforge@gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

Re: use libqmi or .json definition for qmi calls

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 5263 bytes --]

Hi Harald,

Sorry for the late reply, I've been traveling...

> But then, as I iterated in my previous mail, there are plenty of modems
> in solder type (LGA), mPCIe and M.2/NGFF form-factor that expose voice
> capabilities via QMI.  There's a bunch of Sierra Wireless ones (like
> MC7304 & co) as well as EC-20 and EC-25 from Quectel.  I also have some
> Hauwei branded here that do QMI for voice.

If you can get these to work, that'd be nice.  I remember looking at the 
Voice control interface over QMI and it should be fairly simple to write 
a driver for it.

>
> And yes, we will in some way or another interface QMI for voice on at
> least a subset of those modems.  The question is whether we spend time
> on hand-crafting messages [error prone] or use the existing JSON
> description and generated parser/encoders.  I'm not sure which way is
> faster/easier.  If you are fundamentally against code genreation (which
> you are not, AFAICT), we would skip that option.  But if you're open to
> it within the constraints regarding glib and external dependencies, it
> might be an idea to try.

If it were me, I'd just use the JSON description to bootstrap the code 
and then proceed with the 'hand-crafted' option.  I think this is the 
path of least resistance.  We can always add unit tests to minimize the 
errors from this process.  I would expect such an approach to be faster 
overall.

Auto generated code would only make sense if each platform has its own 
(significantly different) dialect of QMI.  But in this case your entire 
premise of using QMI as a baseline to test multiple vendor's products no 
longer holds.

>> Yes, but at least the gobi cards I’ve seen only expose a single
>> network interface.  So multiple contexts are not possible.
>
> I think this has all slightly changed a bit in the last 3+ years when
> the Qualcomm LE stack is used on modern MDM chips.  There you have a
> Linux kernel (and adb, but real glibc and no android userland) inside a
> Cortex-A5 on the modem itself.  In those devices, you can at runtime
> change the configuration/coposition of the USB interface and expose many
> more things that are otherwise not exposed.
>

If you can get this to work, there would be many happy people.  In 
theory it might also be possible to run an extra context over AT/PPP 
channel.  But that would be quite ugly.

>> No such documentation exists.  We had to pick and choose our battles
>> and thus never had the resources for something like that.  I’m happy
>> to host such a matrix in case someone is interested in contributing
>> it.
>
> Well, hosting/maintaining the document (which could just be in git) is
> the easy part.  The more difficult part is collecting information of who
> has actually successfully used which feature on which modem using which
> modem firmware version and which ofono version.

True, hence the resources comment above :)

>
> In the best case, we are able to provide such information based on the
> osmo-gsm-tester in the future.  Our main use case is to test a fixed
> and trusted combination of (ofono + modem + modem firmware) against our
> changing GSM/GPRS/UTMS cellular network stack.  But the same setup (or a
> copy of it) could of course also be used to have a fixed version of the
> GSM/GPRS/UMTS network and test a given [new] ofono version/commit
> against one or multiple modems as part of continuous automation.
>
> So I think there's something for both projects to benefit from.

Definitely!

>> Agreed.  But then I’m a bit confused on what you’re trying to
>> accomplish?
>
> We want to test our GSM/GPRS/UMTS network side protocol stack and
> network elements, preferably with as many different modems as
> possible, but starting with Qualcomm modems with QMI and DIAG as that
> gives us additional insight into what's happening inside the modem-side
> protocol stack.

Okay, the DIAG tidbit makes things a bit more clear now.

> Let me re-iterate that question: Which modem should I buy today for the
> most feature-complete + reliable ofono support?  Please specify an exact
> model number and preferably firmware version.  Then we can buy one of
> these and compare the results of that "known good" vs. whatever we're
> trying on other modems.  That also helps to differentiate while
> debugging between what might be a modem/driver specific issue and a more
> general ofono issue.
>

I don't have an easy answer for you unfortunately.  I can tell you this:
We took oFono with an IFX 6160 modem to a test lab and passed all 
certification tests (Release 8).  plugins/ifx.c is a good example of a 
complete AT command based modem driver.  oFono core has not changed much 
since then besides adding some additional LTE support.

This of course doesn't help you since this device/firmware combination 
is not available.

 From looking at the current set of drivers, the best supported ones in 
my estimation are for Telit and uBlox.

I would also take a look at the Telit xN930 modules.  Their AT command 
spec has a striking similarity to the AT commands being used by the IFX 
driver and is different from the other Telit modules.

Regards,
-Denis