RFC: Ubuntu Touch, MMS, and Provisioning

RFC: Ubuntu Touch, MMS, and Provisioning

[Tony Espy]

[-- Attachment #1: Type: text/plain, Size: 3821 bytes --]

We've been working on MMS support for Ubuntu Touch recently and have run 
into a couple of stumbling blocks, so I have a few questions about the 
current MMS logic in oFono ( we're still 1.12 based ), and in particular 
the provisioning/management of gprs-contexts.

As part of this work, we're planning to switch from using the builtin 
gprs 'provision' plugin, to the new android-provision/apndb plugin ( 
originally written by Simon Busch ) which uses the file apns-conf.xml 
instead of mbpi's serviceproviders.xml.

There are a couple of issues we ran into...

1. How to represent APNs that support multiple usages types?

The "type" attribute in apns-conf.xml is a list vs. the single type 
defined by a gprs_context.

The android-apndb plugin uses TYPE_INTERNET for APNs which support 
multiple types.  In order to properly support MMS, the core code needed 
to be modified to allow the MMS properties to be additionally be set on 
a TYPE_INTERNET context.  Thus if an APN supports both Internet and MMS, 
our DownloadManager can grab the additional MMS properties from the 
context and handle MMS traffic.

Now perhaps I missed something and there is a way to represent a 
combined usage APN ( maybe using TYPE_ANY? ), but I couldn't see how to 
accomplish without changes to the core gprs code.

2. No way to disable core ofono TYPE_MMS network config.

The core gprs_context code has special logic for TYPE_MMS contexts which 
configures the HTTP proxy using networking ioctl requests.  We have an 
external download manager that handles the actual download of content 
from the message center.  As it has logic to handle HTTP proxies 
already, if we use TYPE_MMS, we'd need a way to disable ofono's builtin 
logic.

Would it make sense to add a command-line option to control this behavior?

3. No way to associate additional APN properties with a gprs_context.

apns-conf.xml has many additional APN attributes which don't map 
directly to gprs_context properties ( eg. authtype, mvno_type, ... ).

"types" is especially important, as without it, we can't tell exactly 
which services are supported by the APN ( as the plugin sets the type to 
TYPE_INTERNET ).

So, here's a summary of the changes I made ( again with MMS support as 
one of the main goals ).  Also note, this is WIP, and if there other 
approaches I should consider, I'd appreciate some feedback.

My latest code containing these changes can be found in the following 
github branch.

https://github.com/tonyespy/ofono/tree/mms-new

Note, if it'd be more helpful, I can email the patches directly to the 
list with a "RFC" in the subject line, however I think there's little 
chance that they'd be taken "as is".

1. Modified the core gprs code to allow MMS proxy and message center 
properties to be set on OFONO_GPRS_CONTEXT_TYPE_INTERNET contexts.

2. Modified the android_apndb plugin such that it always creates 
TYPE_INTERNET contexts ( even for standalone MMS APNs, as this allows us 
to bypass the previously mentioned HTTP proxy logic ).

3. Modified the core gprs_context code to allow a new property called 
'ExtraData' to be set during provisioning.  This property is a DBus 
dictionary type ( ie. like 'Setttings' ), and is used by provisioning to 
convey any attributes in apn-confs.xml which don't map directly to ofono 
gprs_contexts.  The current impl doesn't support writes to ExtraData 
from DBus yet.  We may add support to allow a "preferred" attribute ( 
ie. usabe by our Cellular Settings UI ).

If there's a way to accomplish any of these changes via plugins alone, I 
would've gone that route...  Also, if I've missed any other hooks that 
might help us accomplish these changes, I'm welcome to suggestions.

Regards,
/tony

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 2261 bytes --]

Hi Tony,

 > Now perhaps I missed something and there is a way to represent a
> combined usage APN ( maybe using TYPE_ANY? ), but I couldn't see how to
> accomplish without changes to the core gprs code.

oFono is not designed to handle combined usage APNs.  If the APN for MMS 
and Internet is the same, we would simply 'double-activate' the same APN 
using multiple contexts. e.g. you would have an Internet context and an 
MMS context with the same APN.  This of course requires your underlying 
modem driver to be able to use multiple contexts simultaneously (e.g. 
multiple gprs_context atoms present.)

If you want to go through the trouble of adding hybrid behavior, then 
patches are welcome.

>
> 2. No way to disable core ofono TYPE_MMS network config.
>
> The core gprs_context code has special logic for TYPE_MMS contexts which
> configures the HTTP proxy using networking ioctl requests.  We have an
> external download manager that handles the actual download of content
> from the message center.  As it has logic to handle HTTP proxies
> already, if we use TYPE_MMS, we'd need a way to disable ofono's builtin
> logic.
>
> Would it make sense to add a command-line option to control this behavior?

oFono only sets up a route and runs the equivalent of ifconfig.  It 
doesn't do any proxying.  This logic is there because ConnMan explicitly 
ignores contexts of type=mms.  So someone must take care of bringing up 
the interface, assigning IP, etc.

You might also want to check out the mmsd project since that already 
works with oFono.

>
> 3. No way to associate additional APN properties with a gprs_context.
>
> apns-conf.xml has many additional APN attributes which don't map
> directly to gprs_context properties ( eg. authtype, mvno_type, ... ).
>

If you have usecases in mind for some of these properties, feel free to 
suggest extensions to the oFono API.

> "types" is especially important, as without it, we can't tell exactly
> which services are supported by the APN ( as the plugin sets the type to
> TYPE_INTERNET ).

As mentioned above, you'd simply create multiple contexts with the same 
APN and different types if you wanted to accomplish this easily.

Regards,
-Denis

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Slava Monich]

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]


>> 3. No way to associate additional APN properties with a gprs_context.
>>
>> apns-conf.xml has many additional APN attributes which don't map
>> directly to gprs_context properties ( eg. authtype, mvno_type, ... ).
>>
>
> If you have usecases in mind for some of these properties, feel free 
> to suggest extensions to the oFono API.
>

What do you think about extending org.ofono.ConnectionContext API with 
Add/RemoveProperty calls and PropertyAdded/Removed signals to allow 
platform/application specific properties of type string? Use cases 
include marking connections as preferred/default or assigning priorities 
to them, tracking the origin of the context (OTA, local provision or 
manually edited) and who knows what else.

Regards,
-Slava

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 1278 bytes --]

Hi Slava,

On 03/05/2014 03:39 AM, Slava Monich wrote:
>
>>> 3. No way to associate additional APN properties with a gprs_context.
>>>
>>> apns-conf.xml has many additional APN attributes which don't map
>>> directly to gprs_context properties ( eg. authtype, mvno_type, ... ).
>>>
>>
>> If you have usecases in mind for some of these properties, feel free
>> to suggest extensions to the oFono API.
>>
>
> What do you think about extending org.ofono.ConnectionContext API with
> Add/RemoveProperty calls and PropertyAdded/Removed signals to allow
> platform/application specific properties of type string? Use cases
> include marking connections as preferred/default or assigning priorities
> to them, tracking the origin of the context (OTA, local provision or
> manually edited) and who knows what else.
>

There are practical considerations that make this a bad idea, e.g. 
security concerns or the fact that our dbus bindings do not handle 
dynamic properties.  Putting that aside...

Your suggestion would be completely against our three core design 
values, e.g. "Minimal and Complete"; "Consistent" and "Easy to Use".

I'm open to adding additional properties the "old way" if you can argue 
good use cases for them.

Regards,
-Denis

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Slava Monich]

[-- Attachment #1: Type: text/plain, Size: 2274 bytes --]

Hi Denis,

>> What do you think about extending org.ofono.ConnectionContext API with
>> Add/RemoveProperty calls and PropertyAdded/Removed signals to allow
>> platform/application specific properties of type string? Use cases
>> include marking connections as preferred/default or assigning priorities
>> to them, tracking the origin of the context (OTA, local provision or
>> manually edited) and who knows what else.
>>
>
> There are practical considerations that make this a bad idea, e.g. 
> security concerns or the fact that our dbus bindings do not handle 
> dynamic properties.  Putting that aside...
>
> Your suggestion would be completely against our three core design 
> values, e.g. "Minimal and Complete"; "Consistent" and "Easy to Use".
>
> I'm open to adding additional properties the "old way" if you can 
> argue good use cases for them.

A couple of examples.

Suppose, we have a UI that allows the user to switch between "Automatic" 
and "Custom" MMS or GPRS settings. One of the ways to implement that 
would be to create a new context marked as "manual" and allow the user 
to edit it. The old context remains but it's marked as "automatic" or 
whatever. Manual context has a precedence over the automatic one. 
Switching back to automatic means destroying the "manual" context or 
marking it as "disabled". All that stuff gets saved to the ofono 
SIM-specific gprs file so that these context properties don't get lost 
after swapping SIMs.

Suppose, we have a different connection management system, which shows 
the entire list of available access points to the user and allows to 
choose which one to use. The selected context needs to be marked as 
"default" or something. Again, this context property needs to survive 
SIM swap. With a custom context property that's pretty easy to do.

Every project may have requirements that are more or less unique, often 
influenced by the UI design. There's no need to push support for unique 
project specific requirements into the common code, there's nothing here 
to argue about. It's a matter of whether you want to make ofono a bit 
more usable as is or you are fine with it being cloned and heavily 
modified for each particular project.

Regards,
-Slava

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 2036 bytes --]

Hi Slava,

 > A couple of examples.
>
> Suppose, we have a UI that allows the user to switch between "Automatic"
> and "Custom" MMS or GPRS settings. One of the ways to implement that
> would be to create a new context marked as "manual" and allow the user
> to edit it. The old context remains but it's marked as "automatic" or
> whatever. Manual context has a precedence over the automatic one.
> Switching back to automatic means destroying the "manual" context or
> marking it as "disabled". All that stuff gets saved to the ofono
> SIM-specific gprs file so that these context properties don't get lost
> after swapping SIMs.

That sounds awful from a user perspective.  I don't see why you need to 
store these attributes inside oFono.  Just let the user edit the 
existing context and re-run your provisioning application when the user 
makes the decision to switch back to 'automatic'.  Or better yet, get a 
decent provisioning database so that the user isn't asked for this at all.

>
> Suppose, we have a different connection management system, which shows
> the entire list of available access points to the user and allows to
> choose which one to use. The selected context needs to be marked as
> "default" or something. Again, this context property needs to survive
> SIM swap. With a custom context property that's pretty easy to do.

Feel free to suggest something, maybe a Priority attribute.  But then 
again, a better provisioning database or a dedicated provisioning UI 
would be way nicer from a user perspective.

>
> Every project may have requirements that are more or less unique, often
> influenced by the UI design. There's no need to push support for unique
> project specific requirements into the common code, there's nothing here
> to argue about. It's a matter of whether you want to make ofono a bit
> more usable as is or you are fine with it being cloned and heavily
> modified for each particular project.
>

Knock yourself out then :)

Regards,
-Denis

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Marcel Holtmann]

[-- Attachment #1: Type: text/plain, Size: 3999 bytes --]

Hi Slava,

>>> What do you think about extending org.ofono.ConnectionContext API with
>>> Add/RemoveProperty calls and PropertyAdded/Removed signals to allow
>>> platform/application specific properties of type string? Use cases
>>> include marking connections as preferred/default or assigning priorities
>>> to them, tracking the origin of the context (OTA, local provision or
>>> manually edited) and who knows what else.
>>> 
>> 
>> There are practical considerations that make this a bad idea, e.g. security concerns or the fact that our dbus bindings do not handle dynamic properties.  Putting that aside...
>> 
>> Your suggestion would be completely against our three core design values, e.g. "Minimal and Complete"; "Consistent" and "Easy to Use".
>> 
>> I'm open to adding additional properties the "old way" if you can argue good use cases for them.
> 
> A couple of examples.
> 
> Suppose, we have a UI that allows the user to switch between "Automatic" and "Custom" MMS or GPRS settings. One of the ways to implement that would be to create a new context marked as "manual" and allow the user to edit it. The old context remains but it's marked as "automatic" or whatever. Manual context has a precedence over the automatic one. Switching back to automatic means destroying the "manual" context or marking it as "disabled". All that stuff gets saved to the ofono SIM-specific gprs file so that these context properties don't get lost after swapping SIMs.
> 
> Suppose, we have a different connection management system, which shows the entire list of available access points to the user and allows to choose which one to use. The selected context needs to be marked as "default" or something. Again, this context property needs to survive SIM swap. With a custom context property that's pretty easy to do.

I am not even sure what’s your goal here? Are just trying to plain copy the horrible crap that Android puts in front of their users? You do realize that their list of APNs just purely exists since they never figured out on how to do the automatic provisioning properly and user friendly.

I have to second Denis here that time is better spent in creating a proper provision database instead of just copying what AOSP provides. For example many operators are using SPN or IMSI prefixes to clearly identify their MNVO. We made the provisioning a plugin design for exactly that reason so that every product can provide its own database.

You might also think this through once more. What does default actually mean. Can every context have a default tag? Which one do you pick then? Who is making this policy and who is ensuring what is active at what time. oFono is happily supporting multiple Internet contexts and have them all active. Our own modem hardware supports many contexts. We have run this with 4 Internet contexts at the same time (different and same ones).

And I have to note that oFono never establishes contexts by itself. The only thing that oFono does is attach to the GPRS (packet switched) part of network. The APN establishment is a policy enforced by ConnMan for Internet contexts and mmsd for MMS contexts.

The simplest approach we found with our devices is to only provide one Internet context from the user interface. If it is wrong for whatever reason or provisioning can not identify it, then the user gets asked once. If user wants to change it they can go through the APN setup process once more. Worked smoothly for us and is super user friendly.

> Every project may have requirements that are more or less unique, often influenced by the UI design. There's no need to push support for unique project specific requirements into the common code, there's nothing here to argue about. It's a matter of whether you want to make ofono a bit more usable as is or you are fine with it being cloned and heavily modified for each particular project.

Do what we want or we fork your project. Nice :)

Regards

Marcel

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Slava Monich]

[-- Attachment #1: Type: text/plain, Size: 2751 bytes --]

Hi Marcel,
> I am not even sure what’s your goal here? Are just trying to plain copy the horrible crap that Android puts in front of their users? You do realize that their list of APNs just purely exists since they never figured out on how to do the automatic provisioning properly and user friendly.
>
> I have to second Denis here that time is better spent in creating a proper provision database instead of just copying what AOSP provides. For example many operators are using SPN or IMSI prefixes to clearly identify their MNVO. We made the provisioning a plugin design for exactly that reason so that every product can provide its own database.
>
> You might also think this through once more. What does default actually mean. Can every context have a default tag? Which one do you pick then? Who is making this policy and who is ensuring what is active at what time. oFono is happily supporting multiple Internet contexts and have them all active. Our own modem hardware supports many contexts. We have run this with 4 Internet contexts at the same time (different and same ones).
>
> And I have to note that oFono never establishes contexts by itself. The only thing that oFono does is attach to the GPRS (packet switched) part of network. The APN establishment is a policy enforced by ConnMan for Internet contexts and mmsd for MMS contexts.
>
> The simplest approach we found with our devices is to only provide one Internet context from the user interface. If it is wrong for whatever reason or provisioning can not identify it, then the user gets asked once. If user wants to change it they can go through the APN setup process once more. Worked smoothly for us and is super user friendly.

I simply gave you a couple of more or less real life examples as why 
people may want to associate additional information with connection 
contexts and keep it persistent across SIM swaps. It doesn't means that 
I'm actually doing exactly what I have described. And in any case it's 
not my job as a middleware developer to design the UI. But in the end 
when it's decided which settings should be per-SIM and which are global, 
I see ofono which I'm already using and which implements persistent 
per-SIM storage, property change notifications and other niceties and 
wish I could just add a few little properties there.

I thought I suggested a simple, flexible and perfectly backward 
compatible solution which would allow to avoid both a) unnecessary 
forking of ofono and b) public arguments about "horrible Android-like 
crap", "you are doing it all wrong", "you don't really need that" and 
such. And yet let people do what they think is right for their own 
project. Doesn't that sound good?

Regards,
-Slava

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 1129 bytes --]

Hi Slava,

 > not my job as a middleware developer to design the UI. But in the end
> when it's decided which settings should be per-SIM and which are global,
> I see ofono which I'm already using and which implements persistent
> per-SIM storage, property change notifications and other niceties and
> wish I could just add a few little properties there.

The problem is, oFono _is_ in the business of dictating the UI.  The 
entire project was built on that premise and with that philosophy in 
mind.  It was also made to integrate nicely with sister projects 
(ConnMan, BlueZ, mmsd).

>
> I thought I suggested a simple, flexible and perfectly backward
> compatible solution which would allow to avoid both a) unnecessary
> forking of ofono and b) public arguments about "horrible Android-like
> crap", "you are doing it all wrong", "you don't really need that" and
> such. And yet let people do what they think is right for their own
> project. Doesn't that sound good?

You're free to do what you feel is good for your project, as we are free 
to do what we feel is good for ours.

Regards,
-Denis

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Slava Monich]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

Hi Denis,

> Hi Slava,
>
> > not my job as a middleware developer to design the UI. But in the end
>> when it's decided which settings should be per-SIM and which are global,
>> I see ofono which I'm already using and which implements persistent
>> per-SIM storage, property change notifications and other niceties and
>> wish I could just add a few little properties there.
>
> The problem is, oFono _is_ in the business of dictating the UI. The 
> entire project was built on that premise and with that philosophy in 
> mind.  It was also made to integrate nicely with sister projects 
> (ConnMan, BlueZ, mmsd).

It appears that I misunderstood the purpose of the ofono project. I 
thought it was a piece of reusable middleware for building mobile 
telephony applications. Didn't know that it was designed to work with a 
particular UI. That explains a lot actually. Thanks for clearing that up.

Regards,
-Slava

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Tony Espy]

[-- Attachment #1: Type: text/plain, Size: 5625 bytes --]

On 03/05/2014 01:29 PM, Marcel Holtmann wrote:
> Hi Slava,
>
>>>> What do you think about extending org.ofono.ConnectionContext API with
>>>> Add/RemoveProperty calls and PropertyAdded/Removed signals to allow
>>>> platform/application specific properties of type string? Use cases
>>>> include marking connections as preferred/default or assigning priorities
>>>> to them, tracking the origin of the context (OTA, local provision or
>>>> manually edited) and who knows what else.
>>>>
>>>
>>> There are practical considerations that make this a bad idea, e.g. security concerns or the fact that our dbus bindings do not handle dynamic properties.  Putting that aside...
>>>
>>> Your suggestion would be completely against our three core design values, e.g. "Minimal and Complete"; "Consistent" and "Easy to Use".
>>>
>>> I'm open to adding additional properties the "old way" if you can argue good use cases for them.
>>
>> A couple of examples.
>>
>> Suppose, we have a UI that allows the user to switch between "Automatic" and "Custom" MMS or GPRS settings. One of the ways to implement that would be to create a new context marked as "manual" and allow the user to edit it. The old context remains but it's marked as "automatic" or whatever. Manual context has a precedence over the automatic one. Switching back to automatic means destroying the "manual" context or marking it as "disabled". All that stuff gets saved to the ofono SIM-specific gprs file so that these context properties don't get lost after swapping SIMs.
>>
>> Suppose, we have a different connection management system, which shows the entire list of available access points to the user and allows to choose which one to use. The selected context needs to be marked as "default" or something. Again, this context property needs to survive SIM swap. With a custom context property that's pretty easy to do.
>
> I am not even sure what’s your goal here? Are just trying to plain copy the horrible crap that Android puts in front of their users?
> You do realize that their list of APNs just purely exists since they never figured out on how to do the automatic provisioning properly and user friendly.

No, I didn't realize this.  I've generally had positive experiences with 
the Android phones ( mostly Nexus*, except for a G1 ) I've owned.

That said, apns-conf has a lot more content than mbpi, especially with 
regards to MMS support.  There are grand total of 16 MMS APNs defined in 
the latest version of mbpi:

https://git.gnome.org/browse/mobile-broadband-provider-info/

> I have to second Denis here that time is better spent in creating a proper provision database instead of just copying what AOSP provides.

Sure, but how long has mbpi been around for, and why isn't it in better 
shape?  Also, as explained earlier Android supports combined contexts, 
which is not compatible with mbpi's schema.

I'm not against improving mbpi or even working to help create a new 
database.  For us right now, apns-conf is a viable alternative which 
gives us decent provisioning for both Internet and MMS, so we'd like to 
ensure we can support it.

> For example many operators are using SPN or IMSI prefixes to clearly identify their MNVO.
> We made the provisioning a plugin design for exactly that reason so that every product can provide its own database.

This is exactly what we're trying to do, however as my original email 
said, we ran into a few stumbling blocks...

> You might also think this through once more. What does default actually mean. Can every context have a default tag?

As far as I understand it, "default" indicates support for Internet.

Also, please understand that the UI examples that Slava included apply 
to Sailfish's UI, not Ubuntu Touch.

> Which one do you pick then? Who is making this policy and who is ensuring what is active at what time.

We give preference to the first context of type "default" ( for which we 
create an INET context ), unless this has been overriden by the user 
from a settings UI.  I'd considered making my new "ExtraData" propertly 
write-able, so that the UI could add some tag/attribute to denote this.

The other choice of course is to store this meta-data somewhere else... 
which would impact Network Manager, as it's the process responsible for 
activating/deactivating contexts for Touch.

> oFono is happily
> supporting multiple Internet contexts and have them all active. Our own modem hardware supports many contexts.
> We have run this with 4 Internet contexts at the same time (different and same ones).

Understood, as mentioned we need to support multiple contexts for the 
case where a MMS APN is not shared with any other type.

> And I have to note that oFono never establishes contexts by itself. The only thing that oFono does is attach to the GPRS (packet switched) part of network.
> The APN establishment is a policy enforced by ConnMan for Internet contexts and mmsd for MMS contexts.

In our case it's NetworkManager responsible for both, with help from a 
few other system components ( such as our Download Manager ).

> The simplest approach we found with our devices is to only provide one Internet context from the user interface.

Did you expose the MMS context anywhere in your user interface?

> If it is wrong for whatever reason or provisioning can not identify it, then the user gets asked once. If user wants to change it they can go through the APN setup process once more. Worked smoothly for us and is super user friendly.

Makes sense.

Regards,
/tony

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Marcel Holtmann]

[-- Attachment #1: Type: text/plain, Size: 9046 bytes --]

Hi Tony,

>>>>> What do you think about extending org.ofono.ConnectionContext API with
>>>>> Add/RemoveProperty calls and PropertyAdded/Removed signals to allow
>>>>> platform/application specific properties of type string? Use cases
>>>>> include marking connections as preferred/default or assigning priorities
>>>>> to them, tracking the origin of the context (OTA, local provision or
>>>>> manually edited) and who knows what else.
>>>>> 
>>>> 
>>>> There are practical considerations that make this a bad idea, e.g. security concerns or the fact that our dbus bindings do not handle dynamic properties.  Putting that aside...
>>>> 
>>>> Your suggestion would be completely against our three core design values, e.g. "Minimal and Complete"; "Consistent" and "Easy to Use".
>>>> 
>>>> I'm open to adding additional properties the "old way" if you can argue good use cases for them.
>>> 
>>> A couple of examples.
>>> 
>>> Suppose, we have a UI that allows the user to switch between "Automatic" and "Custom" MMS or GPRS settings. One of the ways to implement that would be to create a new context marked as "manual" and allow the user to edit it. The old context remains but it's marked as "automatic" or whatever. Manual context has a precedence over the automatic one. Switching back to automatic means destroying the "manual" context or marking it as "disabled". All that stuff gets saved to the ofono SIM-specific gprs file so that these context properties don't get lost after swapping SIMs.
>>> 
>>> Suppose, we have a different connection management system, which shows the entire list of available access points to the user and allows to choose which one to use. The selected context needs to be marked as "default" or something. Again, this context property needs to survive SIM swap. With a custom context property that's pretty easy to do.
>> 
>> I am not even sure what’s your goal here? Are just trying to plain copy the horrible crap that Android puts in front of their users?
>> You do realize that their list of APNs just purely exists since they never figured out on how to do the automatic provisioning properly and user friendly.
> 
> No, I didn't realize this.  I've generally had positive experiences with the Android phones ( mostly Nexus*, except for a G1 ) I've owned.

it is pretty bad. I have used it in many difference countries and it gets a lot of stuff horrible confused. Fun stuff if you enter Canada and for some reason it decided to configure APNs from Italy after you get out of flight mode.

It also gets re-provisioned once you switch SIM cards which makes this horrible annoying. Something that the iPhone does as well. oFono has this luckily persistent.

> That said, apns-conf has a lot more content than mbpi, especially with regards to MMS support.  There are grand total of 16 MMS APNs defined in the latest version of mbpi:
> 
> https://git.gnome.org/browse/mobile-broadband-provider-info/

The MBPI database is pretty much a joke when it comes to MMS. It is also a joke when it comes to automated provisioning since they stuffed everything in there. So it is hard to make a choice. Then again, the Gnome guys used that for UI based configuration and for that it is not that bad.

For example in one of our devices we used a combined approach. If the APN entries could be clearly identified it gets auto-provisioned by oFono’s plugin. If that failed the UI realizes this and starts a first time setup UI that lets you chose the APN based on the plan information or what not. So the database got used twice.

With the information that oFono provided for the SIM cards MCC and MNC (and the MNC length determination from SIM filesystem is important here), the first time setup agent could be as smart as human possible. So it normally just showed the list of MNVOs and you had to pick one. That was it. After that it was persistently saved by oFono and the user never had to deal with these details again (unless manually triggered by user through settings).

As mentioned in the other email, we realized that the MBPI is not the best database available and we made certain improvements to it already. However the content never got where it needs to be. That is why for oFono it is plugin based. You can just use your own database format and then have your plugin do the job.

Personally I think some simple CSV based format would work just fine. I think that is what Nokia used in the Maemo/MeeGo devices. The real hard job is to go through all the operator information and create a good database. That is the key here. Ignoring that fact is just silly.

>> I have to second Denis here that time is better spent in creating a proper provision database instead of just copying what AOSP provides.
> 
> Sure, but how long has mbpi been around for, and why isn't it in better shape?  Also, as explained earlier Android supports combined contexts, which is not compatible with mbpi's schema.
> 
> I'm not against improving mbpi or even working to help create a new database.  For us right now, apns-conf is a viable alternative which gives us decent provisioning for both Internet and MMS, so we'd like to ensure we can support it.

I can just repeat the fact that we made this plugin based for exactly that reason. Both databases have crap in it that is utterly useless and presented wrongly. We decided to not provide a database of our own since we are not in that business. That is an OEM customization.

>> For example many operators are using SPN or IMSI prefixes to clearly identify their MNVO.
>> We made the provisioning a plugin design for exactly that reason so that every product can provide its own database.
> 
> This is exactly what we're trying to do, however as my original email said, we ran into a few stumbling blocks…

It will not always work. There are some network operators that are just a lost cause. The way we have designed oFono and used auto-provisioning in products is that when oFono can not figure it out, then you need to ask the user. See my comment above.

>> You might also think this through once more. What does default actually mean. Can every context have a default tag?
> 
> As far as I understand it, "default" indicates support for Internet.

In Android terms, yes. Which has no meaning whatsoever anywhere else. As I said, Android has not done a good job with the auto-provisioning. Better than Symbian though ;)

> Also, please understand that the UI examples that Slava included apply to Sailfish's UI, not Ubuntu Touch.
> 
>> Which one do you pick then? Who is making this policy and who is ensuring what is active at what time.
> 
> We give preference to the first context of type "default" ( for which we create an INET context ), unless this has been overriden by the user from a settings UI.  I'd considered making my new "ExtraData" propertly write-able, so that the UI could add some tag/attribute to denote this.
> 
> The other choice of course is to store this meta-data somewhere else... which would impact Network Manager, as it's the process responsible for activating/deactivating contexts for Touch.
> 
>> oFono is happily
>> supporting multiple Internet contexts and have them all active. Our own modem hardware supports many contexts.
>> We have run this with 4 Internet contexts at the same time (different and same ones).
> 
> Understood, as mentioned we need to support multiple contexts for the case where a MMS APN is not shared with any other type.
> 
>> And I have to note that oFono never establishes contexts by itself. The only thing that oFono does is attach to the GPRS (packet switched) part of network.
>> The APN establishment is a policy enforced by ConnMan for Internet contexts and mmsd for MMS contexts.
> 
> In our case it's NetworkManager responsible for both, with help from a few other system components ( such as our Download Manager ).

We decided clearly against involving ConnMan in the MMS context part. It just creates a horrible complicated system. If it is not giving you Internet connectivity, ConnMan stays away from it.

>> The simplest approach we found with our devices is to only provide one Internet context from the user interface.
> 
> Did you expose the MMS context anywhere in your user interface?

The telephony settings application allowed for manual or wizard based configuration. So yes, you were able to see it if you went into telephony settings.

However ConnMan never exposed this and so the networking settings only showed Internet contexts. Keep in mind that the configuration of the Internet context is a telephony/cellular setting detail. ConnMan settings only allowed activated or deactivating the context.

>> If it is wrong for whatever reason or provisioning can not identify it, then the user gets asked once. If user wants to change it they can go through the APN setup process once more. Worked smoothly for us and is super user friendly.
> 
> Makes sense.

Regards

Marcel

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Tony Espy]

[-- Attachment #1: Type: text/plain, Size: 5479 bytes --]

On 03/04/2014 10:13 PM, Denis Kenzior wrote:
> Hi Tony,
>
>  > Now perhaps I missed something and there is a way to represent a
>> combined usage APN ( maybe using TYPE_ANY? ), but I couldn't see how to
>> accomplish without changes to the core gprs code.

Thanks for the reply Denis.

> oFono is not designed to handle combined usage APNs.

Understood...

> If the APN for MMS
> and Internet is the same, we would simply 'double-activate' the same APN
> using multiple contexts. e.g. you would have an Internet context and an
> MMS context with the same APN.  This of course requires your underlying
> modem driver to be able to use multiple contexts simultaneously (e.g.
> multiple gprs_context atoms present.)

OK, applying this logic to the apns-conf db ( I'll get to Marcel's 
second reply focused on this apns shortly ):

1. The provision plugin would need to create separate contexts for each 
type found in an apn's "type" attribute.  This would work when there are 
exact matches between ofono and the apn types ( eg. default=internet, mms ).

2. Regarding multiple contexts supported in the rilmodem.  We're 
planning on doing this anyways, as we need to support the case where a 
MMS apn is stand-alone.

3. For the shared case, the driver context code could be optimized to 
recognize that a context being activated has the same apn as an already 
activated context and just mark the new context activated, borrowing the 
'Settings' from the first.

So this seems do-able, however it does seem like a fair amount of 
overhead, especially when you start to consider apns that are configured 
for more than just "default" & "mms" ( "dun" and "supl" are other types 
supported in apns-conf ).

This precludes creation of apn types that don't match current ofono 
types ( eg. "dun" & "supl" ).  Neither of these are all that important 
to us currently, however if that changes, we'd have a problem supporting 
them without adding new types.


> If you want to go through the trouble of adding hybrid behavior, then
> patches are welcome.

Sure, although I have no doubts this will be easy given some of the 
competing opinions.  ;)

I think the most minimal implementation of shared contexts would be apns 
that support just INET and MMS.  Unless I'm wrong, I think my changes 
that make it allow-able for MessageCenter and MessageProxy to be set on 
a either context type accomplish this.  None of my other changes are 
required for this type of minimal sharing.

I think the larger question of whether this should be more generic has 
yet to be answered...

>> 2. No way to disable core ofono TYPE_MMS network config.
>>
>> The core gprs_context code has special logic for TYPE_MMS contexts which
>> configures the HTTP proxy using networking ioctl requests.  We have an
>> external download manager that handles the actual download of content
>> from the message center.  As it has logic to handle HTTP proxies
>> already, if we use TYPE_MMS, we'd need a way to disable ofono's builtin
>> logic.
>>
>> Would it make sense to add a command-line option to control this
>> behavior?
>
> oFono only sets up a route and runs the equivalent of ifconfig.  It
> doesn't do any proxying.

Got it, I guess I got confused by the function gprs.c::pri_setproxy() 
which sets up a host route for the proxy if configured.  As per Marcel's 
first reply, a proxy is always used for MMS downloads so this will 
always occur.

> This logic is there because ConnMan explicitly
> ignores contexts of type=mms.  So someone must take care of bringing up
> the interface, assigning IP, etc.

Sure, makes sense.  In our case, we'd decided to have NetworkManager 
handle this work for INET or MMS contexts, so again it would be nice if 
this behavior were configurable.

> You might also want to check out the mmsd project since that already
> works with oFono.

We initially had planned to use mmsd, however decided to extend our 
exisitng Download Manager to support downloading MMS content.

>> 3. No way to associate additional APN properties with a gprs_context.
>>
>> apns-conf.xml has many additional APN attributes which don't map
>> directly to gprs_context properties ( eg. authtype, mvno_type, ... ).
>>
>
> If you have usecases in mind for some of these properties, feel free to
> suggest extensions to the oFono API.

Other than "types" ( which has already been discussed ), the only two 
that might make immediate sense are "authtype" ( ie. none, PAP, CHAP, 
both ) and "roaming_protocol"?

I guess from my perspective it felt wrong to just throw away all of 
these extra attributes while provisioning, and thus the dictionary 
property was an approach to preserve them without being too intrusive ( 
although the settings code did complicate things a little ).

Also precedence, as there's additional apn information from mbpi which 
is ignored during provisioning ( <plan>, <usage>, <dns> ).

>> "types" is especially important, as without it, we can't tell exactly
>> which services are supported by the APN ( as the plugin sets the type to
>> TYPE_INTERNET ).
>
> As mentioned above, you'd simply create multiple contexts with the same
> APN and different types if you wanted to accomplish this easily.

Understood, although the resulting number of contexts could result in 
overload to a user if exposed in a settings UI.

Regards,
/tony

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 3710 bytes --]

Hi Tony,

 > 3. For the shared case, the driver context code could be optimized to
> recognize that a context being activated has the same apn as an already
> activated context and just mark the new context activated, borrowing the
> 'Settings' from the first.

It might work.  Generally all contexts have their own interface.  The 
reason is that our data transfer counters gather statistics this way. 
You would have to solve this in some way.

>
> So this seems do-able, however it does seem like a fair amount of
> overhead, especially when you start to consider apns that are configured
> for more than just "default" & "mms" ( "dun" and "supl" are other types
> supported in apns-conf ).

Do you have any idea what these are used for by any chance?

>
> This precludes creation of apn types that don't match current ofono
> types ( eg. "dun" & "supl" ).  Neither of these are all that important
> to us currently, however if that changes, we'd have a problem supporting
> them without adding new types.
>

Cross that bridge when you come to it.

> Sure, although I have no doubts this will be easy given some of the
> competing opinions.  ;)

Things are never easy :)  oFono has been developed with careful 
attention to detail, so touching core parts of the design requires some 
care.  Don't let this be discouraging, oFono maintainers are actually 
rather helpful.

>
> I think the most minimal implementation of shared contexts would be apns
> that support just INET and MMS.  Unless I'm wrong, I think my changes
> that make it allow-able for MessageCenter and MessageProxy to be set on
> a either context type accomplish this.  None of my other changes are
> required for this type of minimal sharing.

There is just one snag.  Applications are not supposed to use 
MessageProxy directly.  Instead they should be using Settings.Proxy. 
This can probably be solved somehow..

>
> Other than "types" ( which has already been discussed ), the only two
> that might make immediate sense are "authtype" ( ie. none, PAP, CHAP,
> both ) and "roaming_protocol"?

I've looked into authtype before.  I think there might be 1 or two 
operators in this world who still require this.  It is a very broken 
design on their part.  If you really, truly need this, it can be added.

What does roaming_protocol do?  Remember, oFono is managing the roaming 
logic internally.

>
> I guess from my perspective it felt wrong to just throw away all of
> these extra attributes while provisioning, and thus the dictionary
> property was an approach to preserve them without being too intrusive (
> although the settings code did complicate things a little ).
>
> Also precedence, as there's additional apn information from mbpi which
> is ignored during provisioning ( <plan>, <usage>, <dns> ).

Just because this information is in the database, doesn't mean it is 
actually required by oFono.

>
>>> "types" is especially important, as without it, we can't tell exactly
>>> which services are supported by the APN ( as the plugin sets the type to
>>> TYPE_INTERNET ).
>>
>> As mentioned above, you'd simply create multiple contexts with the same
>> APN and different types if you wanted to accomplish this easily.
>
> Understood, although the resulting number of contexts could result in
> overload to a user if exposed in a settings UI.

I don't call 2 contexts an overload.  If you have more than 1 of a 
particular type (e.g. internet / mms) then the user should probably get 
to choose one from a list using a UI.  This is where that <plan> and 
<usage> info might be useful; to give some context to the user.

Regards,
-Denis

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Tony Espy]

[-- Attachment #1: Type: text/plain, Size: 8008 bytes --]

On 03/06/2014 12:54 AM, Denis Kenzior wrote:
> Hi Tony,
>
>  > 3. For the shared case, the driver context code could be optimized to
>> recognize that a context being activated has the same apn as an already
>> activated context and just mark the new context activated, borrowing the
>> 'Settings' from the first.
>
> It might work.  Generally all contexts have their own interface.  The
> reason is that our data transfer counters gather statistics this way.
> You would have to solve this in some way.

This isn't always true, especially with rilmodem, as we have yet another 
modem abstraction layer in the picture.

 From the testing I've done, many ril implementations seem to also 
optimize this case and just return the same interface/settings when a 
data call is requested for an APN which already has an active call.

>> So this seems do-able, however it does seem like a fair amount of
>> overhead, especially when you start to consider apns that are configured
>> for more than just "default" & "mms" ( "dun" and "supl" are other types
>> supported in apns-conf ).
>
> Do you have any idea what these are used for by any chance?

As we haven't done much with either type, so my rough understanding of 
the two are:

"supl" is used to indicate an APN that supports a Secure User Plane 
Location server ( used for A-GPS ).

"dun" stands for dial-up networking and I believe is used to indicate an 
APN that supports tethering.

They don't seem to be used all that commonly for stand-alone APNs, and 
"supl" by far is the more prevalent of the two ( by ~10:1 ).

>> This precludes creation of apn types that don't match current ofono
>> types ( eg. "dun" & "supl" ).  Neither of these are all that important
>> to us currently, however if that changes, we'd have a problem supporting
>> them without adding new types.
>>
>
> Cross that bridge when you come to it.
>
>> Sure, although I have no doubts this will be easy given some of the
>> competing opinions.  ;)
>
> Things are never easy :)  oFono has been developed with careful
> attention to detail, so touching core parts of the design requires some
> care.  Don't let this be discouraging, oFono maintainers are actually
> rather helpful.

Undersood, and thanks for the encouragement.

>> I think the most minimal implementation of shared contexts would be apns
>> that support just INET and MMS.  Unless I'm wrong, I think my changes
>> that make it allow-able for MessageCenter and MessageProxy to be set on
>> a either context type accomplish this.  None of my other changes are
>> required for this type of minimal sharing.
>
> There is just one snag.  Applications are not supposed to use
> MessageProxy directly.  Instead they should be using Settings.Proxy.
> This can probably be solved somehow..

I'd assumed that mmsd used 'MessageProxy' directly, but after 
re-examination, I see that it's using "Settings.Proxy".

That said in our case if we'd like NetworkManager to handle the route 
configuration, wouldn't it make sense to use 'MessageProxy' directly?  I 
guess I'm not sure I understand the difference between 'Settings.Proxy' 
and 'MessageProxy' as the former seems to be a copy of the latter.

Also, as both 'MessageProxy' and 'MessageCenter' were always handled 
together in the core gprs code, it made sense to make both available for 
combined contexts.

One last point, it looks like the ofono code presumes that a 
'MessageProxy' value is always in IPV4 number-and-dots notation. Some of 
the proxies defined for APNs in apns-conf contain hostnames.  My guess 
is that this would cause problems for a stand-alone MMS APN, but would 
work with a combined context?

>> Other than "types" ( which has already been discussed ), the only two
>> that might make immediate sense are "authtype" ( ie. none, PAP, CHAP,
>> both ) and "roaming_protocol"?
>
> I've looked into authtype before.  I think there might be 1 or two
> operators in this world who still require this.  It is a very broken
> design on their part.  If you really, truly need this, it can be added.

Well... I'm not sure about the "really, truly" part, but again these 
were examples of other attributes which weren't a stretch to see being 
added as direct ofono properties.

Re: authtype specifically, we've seen at one least modem which chokes 
with our default settings for this parameter.  That said, this may have 
been an operator specific behavior...

Looking at a fairly recent apns-conf, there are close to 500 APNs which 
specify authtype, out of a db of ~2000.  As this is a parameter that our 
rilmodem driver could use when setting up a data call, I would say this 
is useful.

> What does roaming_protocol do?  Remember, oFono is managing the roaming
> logic internally.

I believe this represents the protocol ( ip, ipv6, dual ) to use when 
roaming and activating a context.

>> I guess from my perspective it felt wrong to just throw away all of
>> these extra attributes while provisioning, and thus the dictionary
>> property was an approach to preserve them without being too intrusive (
>> although the settings code did complicate things a little ).
>>
>> Also precedence, as there's additional apn information from mbpi which
>> is ignored during provisioning ( <plan>, <usage>, <dns> ).
>
> Just because this information is in the database, doesn't mean it is
> actually required by oFono.

Understood, however some of these attributes might be interesting to 
other processes using ofono to access the contexts which have been 
provisioned ( and persisted on-disk ).

Again, this is the reason I went ahead and added support for an 
ExtraData dictionary that captures all the extra attributes we can't map 
directly to context properties.  This seemed like the least intrusive 
approach to us.  Are all of these additioanl atrributes absoultely 
critical to us right now?  Probably not, but why throw information away?

Also, as Slava from Jolla pointed out, it also might be nice to use this 
mechanism ( or something similar ) to add some additional private data 
to a context, such as "user-selected", or maybe even a "last-connected".

I hadn't gone this far with my implementation ( ie. allowing writes of 
new keys to ExtraData ), as we'd also discussed using some other 
settings store to write this kind of user preference.  That said, if 
most of the other properties can be updated via the DBus interface, it's 
not a huge stretch...

>>>> "types" is especially important, as without it, we can't tell exactly
>>>> which services are supported by the APN ( as the plugin sets the
>>>> type to
>>>> TYPE_INTERNET ).
>>>
>>> As mentioned above, you'd simply create multiple contexts with the same
>>> APN and different types if you wanted to accomplish this easily.
>>
>> Understood, although the resulting number of contexts could result in
>> overload to a user if exposed in a settings UI.
>
> I don't call 2 contexts an overload.  If you have more than 1 of a
> particular type (e.g. internet / mms) then the user should probably get
> to choose one from a list using a UI.  This is where that <plan> and
> <usage> info might be useful; to give some context to the user.

Sure, if we stick to just Internet/MMS, the number could be managable. 
My point was if an operator defined 3-4 APNS, and each was a multi-usage 
APN ( eg. "default,mms,supl" ), then you'd end up with nine contexts ( 
this assumes creation of a SUPL context type ).

At the moment we're really focused on MMS support, and so at minimum we 
want shared contexts to work for Internet and MMS together. This by 
itself I still believe is a worthy goal, and I think it's do-able with 
minor changes to the core.

That said, before I say much more, I need to re-read and respond to 
Marcel's recent emails.  ;)

I appreciate your feedback and thoughts on this.

Thanks,
/tony

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 4824 bytes --]

Hi Tony,

 >  From the testing I've done, many ril implementations seem to also
> optimize this case and just return the same interface/settings when a
> data call is requested for an APN which already has an active call.

Then that will be something for you to solve then.  oFono actually 
expects a different interface entirely for each active context.

> I'd assumed that mmsd used 'MessageProxy' directly, but after
> re-examination, I see that it's using "Settings.Proxy".
>
> That said in our case if we'd like NetworkManager to handle the route
> configuration, wouldn't it make sense to use 'MessageProxy' directly?  I
> guess I'm not sure I understand the difference between 'Settings.Proxy'
> and 'MessageProxy' as the former seems to be a copy of the latter.

It is due to historical / design reasons.  Using Settings.Proxy would 
allow us to run a local proxy instead of relying on the remote Proxy. 
This allows extra flexibility in the future.  Whether this is still 
something we need can be discussed.

> One last point, it looks like the ofono code presumes that a
> 'MessageProxy' value is always in IPV4 number-and-dots notation. Some of
> the proxies defined for APNs in apns-conf contain hostnames.  My guess
> is that this would cause problems for a stand-alone MMS APN, but would
> work with a combined context?

Since for MMS contexts the only route is via a proxy, DNS resolution is 
impossible.  MessageProxy has to be an ipv4 address.  For combined 
contexts this might be relaxed.

> Re: authtype specifically, we've seen at one least modem which chokes
> with our default settings for this parameter.  That said, this may have
> been an operator specific behavior...
>

3GPP mandates that CHAP is always supported.  In fact the authentication 
is not done against the modem, instead the modem simply takes the 
challenge-response data and forwards it as a part of context activation 
procedure.  Some operators were not following their own standards and 
were not accepting CHAP data, hence the need for authtype.  Last I heard 
this was no longer a problem.  If there are still hold outs, I'd be 
interested in hearing about it.

> Looking at a fairly recent apns-conf, there are close to 500 APNs which
> specify authtype, out of a db of ~2000.  As this is a parameter that our
> rilmodem driver could use when setting up a data call, I would say this
> is useful.

And I would ask that you find at least one example where a context 
activation fails when using a non-default authtype.

> Understood, however some of these attributes might be interesting to
> other processes using ofono to access the contexts which have been
> provisioned ( and persisted on-disk ).

Concrete examples?

>
> Again, this is the reason I went ahead and added support for an
> ExtraData dictionary that captures all the extra attributes we can't map
> directly to context properties.  This seemed like the least intrusive
> approach to us.  Are all of these additioanl atrributes absoultely
> critical to us right now?  Probably not, but why throw information away?
>

Lots of reasons.  Here are the top two:

First, any code that gets accepted gets reviewed and maintained.  You 
are asking us to maintain code that is of dubious value.  So right away 
that is a non-starter.

Second, the API is frozen.  Things can be added, but not removed.  It is 
much easier to add something when you need it than to remove it when you 
realize your ideas were wrong in the first place.

> Also, as Slava from Jolla pointed out, it also might be nice to use this
> mechanism ( or something similar ) to add some additional private data
> to a context, such as "user-selected", or maybe even a "last-connected".
>

Yes it _might_ be nice to someone somewhere ;)  But you still need to 
answer how this is of value to the oFono project?  Remember our core 
design philosophy, Minimal and Complete.  And then ask yourself whether 
adding a particular feature fulfills that philosophy.

>
> Sure, if we stick to just Internet/MMS, the number could be managable.
> My point was if an operator defined 3-4 APNS, and each was a multi-usage
> APN ( eg. "default,mms,supl" ), then you'd end up with nine contexts (
> this assumes creation of a SUPL context type ).

We have a favorite phrase around here, "We don't deal with 
hypotheticals".  Find me an operator that did that, and I will consider 
this case.  Until then it doesn't exist :)

>
> At the moment we're really focused on MMS support, and so at minimum we
> want shared contexts to work for Internet and MMS together. This by
> itself I still believe is a worthy goal, and I think it's do-able with
> minor changes to the core.
>

Agreed ;)

Regards,
-Denis

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Tony Espy]

[-- Attachment #1: Type: text/plain, Size: 7833 bytes --]

On 03/07/2014 12:40 AM, Denis Kenzior wrote:
> Hi Tony,
>
>  >  From the testing I've done, many ril implementations seem to also
>> optimize this case and just return the same interface/settings when a
>> data call is requested for an APN which already has an active call.
>
> Then that will be something for you to solve then.  oFono actually
> expects a different interface entirely for each active context.

My solution was to use a shared context when appropriate and not try to 
start a second data call.  ;)

>> I'd assumed that mmsd used 'MessageProxy' directly, but after
>> re-examination, I see that it's using "Settings.Proxy".
>>
>> That said in our case if we'd like NetworkManager to handle the route
>> configuration, wouldn't it make sense to use 'MessageProxy' directly?  I
>> guess I'm not sure I understand the difference between 'Settings.Proxy'
>> and 'MessageProxy' as the former seems to be a copy of the latter.
>
> It is due to historical / design reasons.  Using Settings.Proxy would
> allow us to run a local proxy instead of relying on the remote Proxy.
> This allows extra flexibility in the future.  Whether this is still
> something we need can be discussed.

OK, thanks for the explanation.

>> One last point, it looks like the ofono code presumes that a
>> 'MessageProxy' value is always in IPV4 number-and-dots notation. Some of
>> the proxies defined for APNs in apns-conf contain hostnames.  My guess
>> is that this would cause problems for a stand-alone MMS APN, but would
>> work with a combined context?
>
> Since for MMS contexts the only route is via a proxy, DNS resolution is
> impossible.  MessageProxy has to be an ipv4 address.

That's what I assumed after looking at the code, but appreciate the 
confirmation.  This might be a good addition to the API documentation ( 
connman-api.txt ).

> For combined contexts this might be relaxed.

And that's certainly what looks to be the case in apns-conf ( ie. 
mmsproxys specified with hostnames only appear for shared contexts, 
although some still use ipv4 addresses. ).

>> Re: authtype specifically, we've seen at one least modem which chokes
>> with our default settings for this parameter.  That said, this may have
>> been an operator specific behavior...
>>
>
> 3GPP mandates that CHAP is always supported.  In fact the authentication
> is not done against the modem, instead the modem simply takes the
> challenge-response data and forwards it as a part of context activation
> procedure.  Some operators were not following their own standards and
> were not accepting CHAP data, hence the need for authtype.  Last I heard
> this was no longer a problem.  If there are still hold outs, I'd be
> interested in hearing about it.

Again, I'm basing this on the contents of a recently shipping apns-conf 
file.

As I'm not able personally to test against these operators I can't 
really offer much proof one way or another as to how compliant they are.

>> Looking at a fairly recent apns-conf, there are close to 500 APNs which
>> specify authtype, out of a db of ~2000.  As this is a parameter that our
>> rilmodem driver could use when setting up a data call, I would say this
>> is useful.
>
> And I would ask that you find at least one example where a context
> activation fails when using a non-default authtype.

As mentioned, that's not possible for me to do directly.

>> Understood, however some of these attributes might be interesting to
>> other processes using ofono to access the contexts which have been
>> provisioned ( and persisted on-disk ).
>
> Concrete examples?

I think I've given enough concrete examples where this kind of 
information in a provisioning db might be useful to an ofono driver 
implementation such as rilmodem, a client of ofono, or both.

That said, if I can't convince you, and it's something that we end up 
requiring, we'll have to carry patches to implement this.

For now I'm willing to forgo trying to convince you that extra data is a 
worthwhile concept, as I'd really like to focus on hybrid MMS/INET support.

>> Again, this is the reason I went ahead and added support for an
>> ExtraData dictionary that captures all the extra attributes we can't map
>> directly to context properties.  This seemed like the least intrusive
>> approach to us.  Are all of these additioanl atrributes absoultely
>> critical to us right now?  Probably not, but why throw information away?
>>
>
> Lots of reasons.  Here are the top two:
>
> First, any code that gets accepted gets reviewed and maintained.  You
> are asking us to maintain code that is of dubious value.  So right away
> that is a non-starter.
>
> Second, the API is frozen.  Things can be added, but not removed.  It is
> much easier to add something when you need it than to remove it when you
> realize your ideas were wrong in the first place.

Well, then let's agree to politely disagree, and move on for now.

Actually though, I do have one question about the API being frozen.  Are 
there any plans to remove the OFONO_API_SUBJECT_TO_CHANGE pre-processor 
logic in plugins.h?

>> Also, as Slava from Jolla pointed out, it also might be nice to use this
>> mechanism ( or something similar ) to add some additional private data
>> to a context, such as "user-selected", or maybe even a "last-connected".
>>
>
> Yes it _might_ be nice to someone somewhere ;)  But you still need to
> answer how this is of value to the oFono project?  Remember our core
> design philosophy, Minimal and Complete.  And then ask yourself whether
> adding a particular feature fulfills that philosophy.

We value the ofono project because it provides a solid, flexible 
framework for providing telephony services to Linux-based devices.

In our case ( and Jolla's ), we felt that this feature helped fulfill 
the mission of shipping Linux-based phones ( based on two very different 
distributions I might add ).

But again, my original email was meant as a RFC, and clearly this 
feature isn't something you're willing to entertain, so I'm willing to 
drop further discussion for the time being.

>> Sure, if we stick to just Internet/MMS, the number could be managable.
>> My point was if an operator defined 3-4 APNS, and each was a multi-usage
>> APN ( eg. "default,mms,supl" ), then you'd end up with nine contexts (
>> this assumes creation of a SUPL context type ).
>
> We have a favorite phrase around here, "We don't deal with
> hypotheticals".  Find me an operator that did that, and I will consider
> this case.  Until then it doesn't exist :)

I'm trying not to deal with hypotheticals either, I'm more focused on 
making sure what we have solid product for OEMs to ship.

As for operator examples, the apns-conf from a Galaxy Nexus (4.3?) 
includes five different APNs for ATT in the US ( mcc/mnc: 310/410 ); 
this includes one APN for the MVNO Straightalk.  I have another newer 
version of apns-conf that has only two defined for the same operator.

The current version of mbpi shipping in Ubuntu has three APNs for ATT/US.

So clearly not hypothetical.

That said, even though the SPN is passed to the provisioning plugin, it 
doesn't appear to be used in the mbpi logic, nor the original version of 
the android_apndb code.

But again, I'd rather focus on whether hybrid INET/MMS contexts could be 
made to work.

>> At the moment we're really focused on MMS support, and so at minimum we
>> want shared contexts to work for Internet and MMS together. This by
>> itself I still believe is a worthy goal, and I think it's do-able with
>> minor changes to the core.
>>
>
> Agreed ;)

Great, we can claim some progress in our discussion then!

Thanks,
/tony

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 1310 bytes --]

Hi Tony,

 >
> Actually though, I do have one question about the API being frozen.  Are
> there any plans to remove the OFONO_API_SUBJECT_TO_CHANGE pre-processor
> logic in plugins.h?

Nope.  The D-Bus API is frozen and we take that seriously.  The internal 
API will likely never be.

> We value the ofono project because it provides a solid, flexible
> framework for providing telephony services to Linux-based devices.

Thanks for the compliment :)

> As for operator examples, the apns-conf from a Galaxy Nexus (4.3?)
> includes five different APNs for ATT in the US ( mcc/mnc: 310/410 );
> this includes one APN for the MVNO Straightalk.  I have another newer
> version of apns-conf that has only two defined for the same operator.

This is where the UI that Marcel and I described comes in.  Ask the user 
which set he wants to use.  Programming in all bazillion contexts is 
pretty pointless.

> That said, even though the SPN is passed to the provisioning plugin, it
> doesn't appear to be used in the mbpi logic, nor the original version of
> the android_apndb code.

That's because MBPI doesn't have any SPN information, nor is it defined 
in the DTD last I checked.  MBPI can / should be improved quite a bit to 
make provisioning more fool proof.

Regards,
-Denis

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Marcel Holtmann]

[-- Attachment #1: Type: text/plain, Size: 2314 bytes --]

Hi Tony,

> We've been working on MMS support for Ubuntu Touch recently and have run into a couple of stumbling blocks, so I have a few questions about the current MMS logic in oFono ( we're still 1.12 based ), and in particular the provisioning/management of gprs-contexts.
> 
> As part of this work, we're planning to switch from using the builtin gprs 'provision' plugin, to the new android-provision/apndb plugin ( originally written by Simon Busch ) which uses the file apns-conf.xml instead of mbpi's serviceproviders.xml.
> 
> There are a couple of issues we ran into...
> 
> 1. How to represent APNs that support multiple usages types?
> 
> The "type" attribute in apns-conf.xml is a list vs. the single type defined by a gprs_context.
> 
> The android-apndb plugin uses TYPE_INTERNET for APNs which support multiple types.  In order to properly support MMS, the core code needed to be modified to allow the MMS properties to be additionally be set on a TYPE_INTERNET context.  Thus if an APN supports both Internet and MMS, our DownloadManager can grab the additional MMS properties from the context and handle MMS traffic.
> 
> Now perhaps I missed something and there is a way to represent a combined usage APN ( maybe using TYPE_ANY? ), but I couldn't see how to accomplish without changes to the core gprs code.
> 
> 2. No way to disable core ofono TYPE_MMS network config.
> 
> The core gprs_context code has special logic for TYPE_MMS contexts which configures the HTTP proxy using networking ioctl requests.  We have an external download manager that handles the actual download of content from the message center.  As it has logic to handle HTTP proxies already, if we use TYPE_MMS, we'd need a way to disable ofono's builtin logic.

you do realize that these are not actually standard HTTP proxies. You are suppose to talk to the MMS Proxy to reach the MMSC. That is how this works. You always go through the MMS Proxy.

If you are mixing Internet proxies with MMS proxies, you end up in a really awkward configuration later down the road. As Denis mentioned we have been using this successfully within mmsd that we wrote. The current design of oFono works just fine by activating the MMS context and then talking to the MMS proxy to reach the MMSC.

Regards

Marcel

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Tony Espy]

[-- Attachment #1: Type: text/plain, Size: 3080 bytes --]

On 03/05/2014 12:57 PM, Marcel Holtmann wrote:
> Hi Tony,
>
>> We've been working on MMS support for Ubuntu Touch recently and have run into a couple of stumbling blocks, so I have a few questions about the current MMS logic in oFono ( we're still 1.12 based ), and in particular the provisioning/management of gprs-contexts.
>>
>> As part of this work, we're planning to switch from using the builtin gprs 'provision' plugin, to the new android-provision/apndb plugin ( originally written by Simon Busch ) which uses the file apns-conf.xml instead of mbpi's serviceproviders.xml.
>>
>> There are a couple of issues we ran into...
>>
>> 1. How to represent APNs that support multiple usages types?
>>
>> The "type" attribute in apns-conf.xml is a list vs. the single type defined by a gprs_context.
>>
>> The android-apndb plugin uses TYPE_INTERNET for APNs which support multiple types.  In order to properly support MMS, the core code needed to be modified to allow the MMS properties to be additionally be set on a TYPE_INTERNET context.  Thus if an APN supports both Internet and MMS, our DownloadManager can grab the additional MMS properties from the context and handle MMS traffic.
>>
>> Now perhaps I missed something and there is a way to represent a combined usage APN ( maybe using TYPE_ANY? ), but I couldn't see how to accomplish without changes to the core gprs code.
>>
>> 2. No way to disable core ofono TYPE_MMS network config.
>>
>> The core gprs_context code has special logic for TYPE_MMS contexts which configures the HTTP proxy using networking ioctl requests.  We have an external download manager that handles the actual download of content from the message center.  As it has logic to handle HTTP proxies already, if we use TYPE_MMS, we'd need a way to disable ofono's builtin logic.
>
> you do realize that these are not actually standard HTTP proxies. You are suppose to talk to the MMS Proxy to reach the MMSC. That is how this works. You always go through the MMS Proxy.

I haven't actually worked directly on the proxy support myself, and had 
assumed this was a standard HTTP proxy.  I will make sure my co-worker 
handling the support in our download manager realizes this, and point 
him at mmsd/connman's GWeb code for reference.

> If you are mixing Internet proxies with MMS proxies, you end up in a really awkward configuration later down the road.

Sorry, no intentions of mixing them.

> As Denis mentioned we have been using this successfully within mmsd that we wrote.

I saw that, we originally did a bunch of work with mmsd, however we 
decided instead to add MMS capability to our ubuntu-download-manager 
used in Touch.

 > The current design of oFono works just fine by activating the
 > MMS context and then talking to the MMS proxy to reach the MMSC.

Sure, although as I mentioned orginally, if we already have an active 
data connection and the apn happens to also support MMS, it seems odd 
not being able to share.

Thanks for jumping in, it's been awhile.  ;)

Regards,
/tony

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Marcel Holtmann]

[-- Attachment #1: Type: text/plain, Size: 4604 bytes --]

Hi Tony,

>>> We've been working on MMS support for Ubuntu Touch recently and have run into a couple of stumbling blocks, so I have a few questions about the current MMS logic in oFono ( we're still 1.12 based ), and in particular the provisioning/management of gprs-contexts.
>>> 
>>> As part of this work, we're planning to switch from using the builtin gprs 'provision' plugin, to the new android-provision/apndb plugin ( originally written by Simon Busch ) which uses the file apns-conf.xml instead of mbpi's serviceproviders.xml.
>>> 
>>> There are a couple of issues we ran into...
>>> 
>>> 1. How to represent APNs that support multiple usages types?
>>> 
>>> The "type" attribute in apns-conf.xml is a list vs. the single type defined by a gprs_context.
>>> 
>>> The android-apndb plugin uses TYPE_INTERNET for APNs which support multiple types.  In order to properly support MMS, the core code needed to be modified to allow the MMS properties to be additionally be set on a TYPE_INTERNET context.  Thus if an APN supports both Internet and MMS, our DownloadManager can grab the additional MMS properties from the context and handle MMS traffic.
>>> 
>>> Now perhaps I missed something and there is a way to represent a combined usage APN ( maybe using TYPE_ANY? ), but I couldn't see how to accomplish without changes to the core gprs code.
>>> 
>>> 2. No way to disable core ofono TYPE_MMS network config.
>>> 
>>> The core gprs_context code has special logic for TYPE_MMS contexts which configures the HTTP proxy using networking ioctl requests.  We have an external download manager that handles the actual download of content from the message center.  As it has logic to handle HTTP proxies already, if we use TYPE_MMS, we'd need a way to disable ofono's builtin logic.
>> 
>> you do realize that these are not actually standard HTTP proxies. You are suppose to talk to the MMS Proxy to reach the MMSC. That is how this works. You always go through the MMS Proxy.
> 
> I haven't actually worked directly on the proxy support myself, and had assumed this was a standard HTTP proxy.  I will make sure my co-worker handling the support in our download manager realizes this, and point him at mmsd/connman's GWeb code for reference.
> 
>> If you are mixing Internet proxies with MMS proxies, you end up in a really awkward configuration later down the road.
> 
> Sorry, no intentions of mixing them.
> 
>> As Denis mentioned we have been using this successfully within mmsd that we wrote.
> 
> I saw that, we originally did a bunch of work with mmsd, however we decided instead to add MMS capability to our ubuntu-download-manager used in Touch.

feels weird if a download manager handles delivery receipts and read receipts. Especially since there are so many MMS specific header in the HTTP request.

> > The current design of oFono works just fine by activating the
> > MMS context and then talking to the MMS proxy to reach the MMSC.
> 
> Sure, although as I mentioned orginally, if we already have an active data connection and the apn happens to also support MMS, it seems odd not being able to share.

If you have an active data connection you can just compare the TYPE_MMS context details (APN, username, passed) with the TYPE_INTERNET context ones and if they match, you choose not to activate the MMS one and just use the Internet one. As I said in the other emails, oFono is activating contexts only on request. So either ConnMan requests the Internet context or mmsd requests the MMS context. oFono is not in the business of magically activating contexts. It only attaches to the GPRS part of the network.

The only problem with the Internet context usage for MMS is that you now also have to talk to ConnMan to figure out when the IP part of the context is successfully configured. And here is where this gets complicated. And active context does not mean it is actively in use. You might actually use your WiFi and have an activated Internet context, but neither the IP or routing is set. Or in same case you are just on WiFi and have no Internet context activated. As you can see this can get extremely complex.

We opted for just providing a MMS context type and an Internet context type. If the APN happens to be the same (not always the case btw.) then you just double activate that context. Send or receive your MMS and deactivate it. We have used this just fine. And in that case oFono does all the heavy lifting for you. You just need to talk to the MMS Proxy address that has been configured. That is it.

Regards

Marcel

Re: RFC: Ubuntu Touch, MMS, and Provisioning

[Tony Espy]

[-- Attachment #1: Type: text/plain, Size: 6605 bytes --]

On 03/05/2014 09:41 PM, Marcel Holtmann wrote:
> Hi Tony,
>
>>>> We've been working on MMS support for Ubuntu Touch recently and have run into a couple of stumbling blocks, so I have a few questions about the current MMS logic in oFono ( we're still 1.12 based ), and in particular the provisioning/management of gprs-contexts.
>>>>
>>>> As part of this work, we're planning to switch from using the builtin gprs 'provision' plugin, to the new android-provision/apndb plugin ( originally written by Simon Busch ) which uses the file apns-conf.xml instead of mbpi's serviceproviders.xml.
>>>>
>>>> There are a couple of issues we ran into...
>>>>
>>>> 1. How to represent APNs that support multiple usages types?
>>>>
>>>> The "type" attribute in apns-conf.xml is a list vs. the single type defined by a gprs_context.
>>>>
>>>> The android-apndb plugin uses TYPE_INTERNET for APNs which support multiple types.  In order to properly support MMS, the core code needed to be modified to allow the MMS properties to be additionally be set on a TYPE_INTERNET context.  Thus if an APN supports both Internet and MMS, our DownloadManager can grab the additional MMS properties from the context and handle MMS traffic.
>>>>
>>>> Now perhaps I missed something and there is a way to represent a combined usage APN ( maybe using TYPE_ANY? ), but I couldn't see how to accomplish without changes to the core gprs code.
>>>>
>>>> 2. No way to disable core ofono TYPE_MMS network config.
>>>>
>>>> The core gprs_context code has special logic for TYPE_MMS contexts which configures the HTTP proxy using networking ioctl requests.  We have an external download manager that handles the actual download of content from the message center.  As it has logic to handle HTTP proxies already, if we use TYPE_MMS, we'd need a way to disable ofono's builtin logic.
>>>
>>> you do realize that these are not actually standard HTTP proxies. You are suppose to talk to the MMS Proxy to reach the MMSC. That is how this works. You always go through the MMS Proxy.
>>
>> I haven't actually worked directly on the proxy support myself, and had assumed this was a standard HTTP proxy.  I will make sure my co-worker handling the support in our download manager realizes this, and point him at mmsd/connman's GWeb code for reference.
>>
>>> If you are mixing Internet proxies with MMS proxies, you end up in a really awkward configuration later down the road.
>>
>> Sorry, no intentions of mixing them.
>>
>>> As Denis mentioned we have been using this successfully within mmsd that we wrote.
>>
>> I saw that, we originally did a bunch of work with mmsd, however we decided instead to add MMS capability to our ubuntu-download-manager used in Touch.
>
> feels weird if a download manager handles delivery receipts and read receipts. Especially since there are so many MMS specific header in the HTTP request.

>>> The current design of oFono works just fine by activating the
>>> MMS context and then talking to the MMS proxy to reach the MMSC.
>>
>> Sure, although as I mentioned orginally, if we already have an active data connection and the apn happens to also support MMS, it seems odd not being able to share.
>
> If you have an active data connection you can just compare the TYPE_MMS context details (APN, username, passed)
> with the TYPE_INTERNET context ones and if they match, you choose not to activate the MMS one and just use the
> Internet one.

Sure, this is along the lines of what Denis suggested, however it does 
seem like a lot of overhead, when in theory the addition of MMS 
properties to a TYPE_INTERNET context would allow it be used for MMS.

> As I said in the other emails, oFono is activating contexts only on request.

Ack

> So either ConnMan requests theInternet context or mmsd requests the MMS context. oFono is not in the 
business of
> magically activating contexs. It only attaches to the GPRS part of the network.

I'm not proposing any kind of auto-activation either.  In our case it's 
NetworkManager that's responsible for activating one or more contexts 
for Internet and MMS on-demand when needed.

Again this is also dependent on our disabling ( probaly via a configure 
option ) the ofono MMS IP addr/static route code.  I will discuss this 
again with our NM maintainer though,

> The only problem with the Internet context usage for MMS is that you now also have to talk to ConnMan to figure out when the IP part of the context is successfully configured.

Well again, as we have Network Manager handling context activation, I 
don't think is an issue for us.

That said, in a default configuration where ConnMan is being used, 
couldn't ofono still setup the static route to the mms proxy if the 
context was combined, and let ConnMan do the rest?

> And here is where this gets complicated. And active context does not mean it is actively in use. You might actually use your WiFi and have an activated Internet context, but neither the IP or routing is set.
> Or in same case you are just on WiFi and have no Internet context activated.

If the Internet context is not activated due to insufficient signal, or 
mobile data is disabled, then all bets are off for MMS, and even we 
tried to activate an independent context in this case, it would fail.

If the Internet context is active, but WiFi was subsequently activated 
and is set as the default route, the static route to the MMS proxy 
should still be available for MMS messages.

I will review again internally to ensure we've covered our bases here.

> As you can see this can get extremely complex.

No argument there.

> We opted for just providing a MMS context type and an Internet context type. If the APN happens to be the same (not always the case btw.) then you just double activate that context. Send or receive your MMS and deactivate it.
> We have used this just fine. And in that case oFono does all the heavy lifting for you. You just need to talk to the MMS Proxy address that has been configured. That is it.'

As explained in a previous reply to Denis, double-activating the same 
context may not result in a new interface when talking to a rild 
implementation.

If we were to stay with this default behavior ( which I still would like 
to avoid by use of shared MMS/INET contexts ), we'd have to implement 
logic discussed previously ( ie. check for an already active context 
with the same APN/username/pw in the rilmodem ).

Thanks again for your comments/feedback Marcel.

Regards,
/tony